Cgroup 用法

时间  2020-01-18
标签 cgroup 用法 繁體版
原文   原文链接

实在懒得写了，cgroup转一篇，Cgroup 用法，写得很详细。对cgroup子系统的分析可参考下面两篇。html

cgroup 分析之CPU和内存部分node

cgroup 子系统之 net_cls 和 net_priogit

介绍docker的的过程当中，提到lxc利用cgroup来提供资源的限额和控制，本文主要介绍cgroup的用法和操做命令，主要内容来自github

[1]https://access.redhat.com/sit...sql

[2]https://www.kernel.org/doc/Do...docker

cgroup

cgroup的功能在于将一台计算机上的资源(CPU,memory, network)进行分片，来防止进程间不利的资源抢占。centos

Terminologyapi

  • cgroup - 关联一组task和一组subsystem的配置参数。一个task对应一个进程, cgroup是资源分片的最小单位。
  • subsystem - 资源管理器，一个subsystem对应一项资源的管理，如 cpu, cpuset, memory等
  • hierarchy - 关联一个到多个subsystem和一组树形结构的cgroup. 和cgroup不一样，hierarchy包含的是可管理的subsystem而非具体参数

因而可知，cgroup对资源的管理是一个树形结构，相似进程。网络

相同点 - 分层结构，子进程/cgroup继承父进程/cgroupapp

不一样点 - 进程是一个单根树状结构(pid=0为根)，而cgroup总体来看是一个多树的森林结构(hierarchy为根)。

一个典型的hierarchy挂载目录以下

/cgroup/
├── blkio                           <--------------- hierarchy/root cgroup                   
│   ├── blkio.io_merged             <--------------- subsystem parameter
... ...
│   ├── blkio.weight
│   ├── blkio.weight_device
│   ├── cgroup.event_control
│   ├── cgroup.procs
│   ├── lxc                         <--------------- cgroup
│   │   ├── blkio.io_merged         <--------------- subsystem parameter
│   │   ├── blkio.io_queued
... ... ...
│   │   └── tasks                   <--------------- task list
│   ├── notify_on_release
│   ├── release_agent
│   └── tasks
...

subsystem列表

RHEL/centos支持的subsystem以下

  • blkio — 块存储配额 >> this subsystem sets limits on input/output access to and from block devices such as physical drives (disk, solid state, USB, etc.).
  • cpu — CPU时间分配限制 >> this subsystem uses the scheduler to provide cgroup tasks access to the CPU.
  • cpuacct — CPU资源报告 >> this subsystem generates automatic reports on CPU resources used by tasks in a cgroup.
  • cpuset — CPU绑定限制 >> this subsystem assigns individual CPUs (on a multicore system) and memory nodes to tasks in a cgroup.
  • devices — 设备权限限制 >> this subsystem allows or denies access to devices by tasks in a cgroup.
  • freezer — cgroup中止/恢复 >> this subsystem suspends or resumes tasks in a cgroup.
  • memory — 内存限制 >> this subsystem sets limits on memory use by tasks in a cgroup, and generates automatic reports on memory resources used by those tasks.
  • net_cls — 配合tc进行网络限制 >> this subsystem tags network packets with a class identifier (classid) that allows the Linux traffic controller (tc) to identify packets originating from a particular cgroup task.
  • net_prio — 网络设备优先级 >> this subsystem provides a way to dynamically set the priority of network traffic per network interface.
  • ns — 资源命名空间限制 >> the namespace subsystem.

cgroup操做准则与方法

操做准则

1.一个hierarchy能够有多个 subsystem (mount 的时候hierarchy能够attach多个subsystem)

A single hierarchy can have one or more subsystems attached to it.

eg.

mount -t cgroup -o cpu,cpuset,memory cpu_and_mem /cgroup/cpu_and_mem

cgroup-rule1

2.一个已经被挂载的 subsystem 只能被再次挂载在一个空的 hierarchy 上 (已经mount一个subsystem的hierarchy不能挂载一个已经被其它hierarchy挂载的subsystem)

Any single subsystem (such as cpu) cannot be attached to more than one hierarchy if one of those hierarchies has a different subsystem attached to it already.

cgroup-rule2

3.每一个task只能在一同个hierarchy的惟一一个cgroup里(不能在同一个hierarchy下有超过一个cgroup的tasks里同时有这个进程的pid)

Each time a new hierarchy is created on the systems, all tasks on the system are initially members of the default cgroup of that hierarchy, which is known as the root cgroup. For any single hierarchy you create, each task on the system can be a member of exactly onecgroup in that hierarchy. A single task may be in multiple cgroups, as long as each of those cgroups is in a different hierarchy. As soon as a task becomes a member of a second cgroup in the same hierarchy, it is removed from the first cgroup in that hierarchy. At no time is a task ever in two different cgroups in the same hierarchy.

cgroup-rule3

4.子进程在被fork出时自动继承父进程所在cgroup，可是fork以后就能够按需调整到其余cgroup

Any process (task) on the system which forks itself creates a child task. A child task automatically inherits the cgroup membership of its parent but can be moved to different cgroups as needed. Once forked, the parent and child processes are completely independent.

cgroup-rule4

5.其它

  • 限制一个task的惟一方法就是将其加入到一个cgroup的task里
  • 多个subsystem能够挂载到一个hierarchy里, 而后经过不一样的cgroup中的subsystem参数来对不一样的task进行限额
  • 若是一个hierarchy有太多subsystem，能够考虑重构 - 将subsystem挂到独立的hierarchy; 相应的, 能够将多个hierarchy合并成一个hierarchy
  • 由于能够只挂载少许subsystem, 能够实现只对task单个方面的限额; 同时一个task能够被加到多个hierarchy中，从而实现对多个资源的控制

操做方法

1.挂载subsystem

  • 利用cgconfig服务及其配置文件 /etc/cgconfig.conf - 服务启动时自动挂载

    subsystem = /cgroup/hierarchy;

  • 命令行操做

    mount -t cgroup -o subsystems name /cgroup/name

    取消挂载

    umount /cgroup/name

    eg. 挂载 cpuset, cpu, cpuacct, memory 4个subsystem到/cgroup/cpu_and_mem 目录(hierarchy)

mount {
      cpuset  = /cgroup/cpu_and_mem;
      cpu    = /cgroup/cpu_and_mem;
      cpuacct = /cgroup/cpu_and_mem;
      memory  = /cgroup/cpu_and_mem;
  }

or

mount -t cgroup -o remount,cpu,cpuset,memory cpu_and_mem /cgroup/cpu_and_mem

2. 新建/删除 cgroup

  • 利用cgconfig服务及其配置文件 /etc/cgconfig.conf - 服务启动时自动挂载

    group <name> {
    [<permissions>]    <controller> {  <param name> = <param value>;
        …
    }
    …
}

  • 命令行操做

    • 新建1 cgcreate -t uid:gid -a uid:gid -g subsystems:path
    • 新建2 mkdir /cgroup/hierarchy/name/child_name
    • 删除1 cgdelete subsystems:path (使用 -r 递归删除)
    • 删除2 rm -rf /cgroup/hierarchy/name/child_name (cgconfig service not running)

3. 权限管理

  • 利用cgconfig服务及其配置文件 /etc/cgconfig.conf - 服务启动时自动挂载

    perm {
    task {
        uid = <task user>;
        gid = <task group>;
    }
    admin {
      uid = <admin name>;
      gid = <admin group>;
    }
}
  • 命令行操做 chown

    eg.

group daemons {
      cpuset {
          cpuset.mems = 0;
          cpuset.cpus = 0;
      }
  }
  group daemons/sql {
      perm {
          task {
              uid = root;
              gid = sqladmin;
          } admin {
              uid = root;
              gid = root;
          }
      }
      cpuset {
          cpuset.mems = 0;
          cpuset.cpus = 0;
      }
  }

or

~]$ mkdir -p /cgroup/red/daemons/sql
  ~]$ chown root:root /cgroup/red/daemons/sql/*
  ~]$ chown root:sqladmin /cgroup/red/daemons/sql/tasks
  ~]$ echo 0 > /cgroup/red/daemons/cpuset.mems
  ~]$ echo 0 > /cgroup/red/daemons/cpuset.cpus
  ~]$ echo 0 > /cgroup/red/daemons/sql/cpuset.mems
  ~]$ echo 0 > /cgroup/red/daemons/sql/cpuset.cpus

4. cgroup参数设定

  • 命令行1 cgset -r parameter=value path_to_cgroup
  • 命令行2 cgset --copy-from path_to_source_cgroup path_to_target_cgroup
  • 文件 echo value > path_to_cgroup/parameter

    eg.

cgset -r cpuset.cpus=0-1 group1
  cgset --copy-from group1/ group2/
  echo 0-1 > /cgroup/cpuset/group1/cpuset.cpus

5. 添加task

  • 命令行添加进程 cgclassify -g subsystems:path_to_cgroup pidlist
  • 文件添加进程 echo pid > path_to_cgroup/tasks
  • 在cgroup中启动进程 cgexec -g subsystems:path_to_cgroup command arguments
  • 在cgroup中启动服务 echo 'CGROUP_DAEMON="subsystem:control_group"' >> /etc/sysconfig/

  • 利用cgrulesengd服务初始化，在配置文件/etc/cgrules.conf中

    user<:command> subsystems control_group

其中:
+用户user的全部进程的subsystems限制的group为control_group
+<:command>是可选项，表示对特定命令实行限制
+user能够用@group表示对特定的 usergroup 而非user
+能够用*表示所有
+%表示和前一行的该项相同

    eg.

cgclassify -g cpu,memory:group1 1701 1138
  echo -e "1701\n1138" |tee -a /cgroup/cpu/group1/tasks /cgroup/memory/group1/tasks
  cgexec -g cpu:group1 lynx http://www.redhat.com
  sh -c "echo \$$ > /cgroup/lab1/group1/tasks && lynx http://www.redhat.com"

经过/etc/cgrules.conf 对特定服务限制

maria          devices        /usergroup/staff
  maria:ftp      devices        /usergroup/staff/ftp
  @student       cpu,memory     /usergroup/student/
  %              memory         /test2/

6. 其余

  • cgsnapshot会根据当前cgroup状况生成/etc/cgconfig.conf文件内容

    gsnapshot  [-s] [-b FILE] [-w FILE] [-f FILE] [controller]
  -b, --blacklist=FILE  Set the blacklist configuration file (default /etc/cgsnapshot_blacklist.conf)
  -f, --file=FILE       Redirect the output to output_file
  -s, --silent          Ignore all warnings
  -t, --strict          Don't show the variables which are not on the whitelist
  -w, --whitelist=FILE  Set the whitelist configuration file (don't used by default)

  • 查看进程在哪一个cgroup

    ps -O cgroup
或
cat /proc/<PID>/cgroup

  • 查看subsystem mount状况

    cat /proc/cgroups

lssubsys -m <subsystems>
  • 查看cgroup lscgroup

  • 查看cgroup参数值

    cgget -r parameter list_of_cgroups
cgget -g <controllers>:<path>
  • cgclear删除hierarchy极其全部cgroup
  • 事件通知API - 目前只支持memory.oom_control

  • 更多

    • man 1 cgclassify — the cgclassify command is used to move running tasks to one or more cgroups.
    • man 1 cgclear — the cgclear command is used to delete all cgroups in a hierarchy.
    • man 5 cgconfig.conf — cgroups are defined in the cgconfig.conf file.
    • man 8 cgconfigparser — the cgconfigparser command parses the cgconfig.conf file and mounts hierarchies.
    • man 1 cgcreate — the cgcreate command creates new cgroups in hierarchies.
    • man 1 cgdelete — the cgdelete command removes specified cgroups.
    • man 1 cgexec — the cgexec command runs tasks in specified cgroups.
    • man 1 cgget — the cgget command displays cgroup parameters.
    • man 1 cgsnapshot — the cgsnapshot command generates a configuration file from existing subsystems.
    • man 5 cgred.conf — cgred.conf is the configuration file for the cgred service.
    • man 5 cgrules.conf — cgrules.conf contains the rules used for determining when tasks belong to certain cgroups.
    • man 8 cgrulesengd — the cgrulesengd service distributes tasks to cgroups.
    • man 1 cgset — the cgset command sets parameters for a cgroup.
    • man 1 lscgroup — the lscgroup command lists the cgroups in a hierarchy.
    • man 1 lssubsys — the lssubsys command lists the hierarchies containing the specified subsystems.

subsystem配置

1. blkio - BLOCK IO限额

  • common

    • blkio.reset_stats - 重置统计信息，写int到此文件
    • blkio.time - 统计cgroup对设备的访问时间 - device_types:node_numbers milliseconds
    • blkio.sectors - 统计cgroup对设备扇区访问数量 - device_types:node_numbers sector_count
    • blkio.avg_queue_size - 统计平均IO队列大小(须要CONFIG_DEBUG_BLK_CGROUP=y)
    • blkio.group_wait_time - 统计cgroup等待总时间(须要CONFIG_DEBUG_BLK_CGROUP=y, 单位ns)
    • blkio.empty_time - 统计cgroup无等待io总时间(须要CONFIG_DEBUG_BLK_CGROUP=y, 单位ns)
    • blkio.idle_time - reports the total time (in nanoseconds — ns) the scheduler spent idling for a cgroup in anticipation of a better request than those requests already in other queues or from other groups.
    • blkio.dequeue - 此cgroup IO操做被设备dequeue次数(须要CONFIG_DEBUG_BLK_CGROUP=y) - device_types:node_numbers number
    • blkio.io_serviced - 报告CFQ scheduler统计的此cgroup对特定设备的IO操做(read, write, sync, or async)次数 - device_types:node_numbers operation number
    • blkio.io_service_bytes - 报告CFQ scheduler统计的此cgroup对特定设备的IO操做(read, write, sync, or async)数据量 - device_types:node_numbers operation bytes
    • blkio.io_service_time - 报告CFQ scheduler统计的此cgroup对特定设备的IO操做(read, write, sync, or async)时间(单位ns) - device_types:node_numbers operation time
    • blkio.io_wait_time - 此cgroup对特定设备的特定操做(read, write, sync, or async)的等待时间(单位ns) - device_types:node_numbers operation time
    • blkio.io_merged - 此cgroup的BIOS requests merged into IO请求的操做(read, write, sync, or async)的次数 - number operation
    • blkio.io_queued - 此cgroup的queued IO 操做(read, write, sync, or async)的请求次数 - number operation

  • Proportional weight division 策略 - 按比例分配block io资源

    • blkio.weight - 100-1000的相对权重，会被blkio.weight_device的特定设备权重覆盖
    • blkio.weight_device - 特定设备的权重 - device_types:node_numbers weight

  • I/O throttling (Upper limit) 策略 - 设定IO操做上限

    • 每秒读/写数据上限
      blkio.throttle.read_bps_device - device_types:node_numbers bytes_per_second blkio.throttle.write_bps_device - device_types:node_numbers bytes_per_second
    • 每秒读/写操做次数上限
      blkio.throttle.read_iops_device - device_types:node_numbers operations_per_second blkio.throttle.write_iops_device - device_types:node_numbers operations_per_second
    • 每秒具体操做(read, write, sync, or async)的控制 blkio.throttle.io_serviced - device_types:node_numbers operation operations_per_second blkio.throttle.io_service_bytes - device_types:node_numbers operation bytes_per_second

2. cpu - CPU使用时间限额

  • CFS(Completely Fair Scheduler)策略 - CPU最大资源限制

    • cpu.cfs_period_us, cpu.cfs_quota_us - 必选 - 两者配合，前者规定时间周期(微秒)后者规定cgroup最多可以使用时间(微秒)，实现task对单个cpu的使用上限(cfs_quota_us是cfs_period_us的两倍便可限定在双核上彻底使用)。
    • cpu.stat - 记录cpu统计信息，包含 nr_periods（经历了几个cfs_period_us）, nr_throttled (cgroup里的task被限制了几回), throttled_time (cgroup里的task被限制了多少纳秒)
    • cpu.shares - 可选 - cpu轮转权重的相对值

  • RT(Real-Time scheduler)策略 - CPU最小资源限制

    • cpu.rt_period_us, cpu.rt_runtime_us

      两者配合使用规定cgroup里的task每cpu.rt_period_us(微秒)必然会执行cpu.rt_runtime_us(微秒)

3. cpuacct - CPU资源报告

  • cpuacct.usage - cgroup中全部task的cpu使用时长(纳秒)
  • cpuacct.stat - cgroup中全部task的用户态和内核态分别使用cpu的时长
  • cpuacct.usage_percpu - cgroup中全部task使用每一个cpu的时长

4. cpuset - CPU绑定

  • cpuset.cpus - 必选 - cgroup可以使用的cpu，如0-2,16表明 0,1,2,16这4个cpu
  • cpuset.mems - 必选 - cgroup可以使用的memory node
  • cpuset.memory_migrate - 可选 - 当cpuset.mems变化时page上的数据是否迁移, default 0
  • cpuset.cpu_exclusive - 可选 - 是否独占cpu， default 0
  • cpuset.mem_exclusive - 可选 - 是否独占memory，default 0
  • cpuset.mem_hardwall - 可选 - cgroup中task的内存是否隔离， default 0
  • cpuset.memory_pressure - 可选 - a read-only file that contains a running average of the memory pressure created by the processes in this cpuset
  • cpuset.memory_pressure_enabled - 可选 - cpuset.memory_pressure开关，default 0
  • cpuset.memory_spread_page - 可选 - contains a flag (0 or 1) that specifies whether file system buffers should be spread evenly across the memory nodes allocated to this cpuset， default 0
  • cpuset.memory_spread_slab - 可选 - contains a flag (0 or 1) that specifies whether kernel slab caches for file input/output operations should be spread evenly across the cpuset， default 0
  • cpuset.sched_load_balance - 可选 - cgroup的cpu压力是否会被平均到cpu set中的多个cpu, default 1

  • cpuset.sched_relax_domain_level - 可选 - cpuset.sched_load_balance的策略

    • -1 = Use the system default value for load balancing
    • 0 = Do not perform immediate load balancing; balance loads only periodically
    • 1 = Immediately balance loads across threads on the same core
    • 2 = Immediately balance loads across cores in the same package
    • 3 = Immediately balance loads across CPUs on the same node or blade
    • 4 = Immediately balance loads across several CPUs on architectures with non-uniform memory access (NUMA)
    • 5 = Immediately balance loads across all CPUs on architectures with NUMA

5. device - cgoup的device限制

  • 设备黑/白名单

    • devices.allow - 容许名单
    • devices.deny - 禁止名单
    • 语法 - type device_types:node_numbers access type - b (块设备) c (字符设备) a (所有设备) access - r 读 w 写 m 建立
  • devices.list - 报告

6. freezer - 暂停/恢复 cgroup的限制

  • 不能出如今root目录下
  • freezer.state - FROZEN 中止 FREEZING 正在中止 THAWED 恢复

7. memory - 内存限制

  • memory.usage_in_bytes - 报告内存限制byte
  • memory.memsw.usage_in_bytes - 报告cgroup中进程当前所用内存+swap空间
  • memory.max_usage_in_bytes - 报告cgoup中的最大内存使用
  • memory.memsw.max_usage_in_bytes - 报告最大使用到的内存+swap
  • memory.limit_in_bytes - cgroup - 最大内存限制，单位k,m,g. -1表明取消限制
  • memory.memsw.limit_in_bytes - 最大内存+swap限制，单位k,m,g. -1表明取消限制
  • memory.failcnt - 报告达到最大容许内存的次数
  • memory.memsw.failcnt - 报告达到最大容许内存+swap的次数
  • memory.force_empty - 设为0且无task时，清除cgroup的内存页
  • memory.swappiness - 换页策略，60基准，小于60下降换出机率，大于60增长换出机率
  • memory.use_hierarchy - 是否影响子group
  • memory.oom_control - 0 enabled，当oom发生时kill掉进程

  • memory.stat - 报告cgroup限制状态

    • cache - page cache, including tmpfs (shmem), in bytes
    • rss - anonymous and swap cache, not including tmpfs (shmem), in bytes
    • mapped_file - size of memory-mapped mapped files, including tmpfs (shmem), in bytes
    • pgpgin - number of pages paged into memory
    • pgpgout - number of pages paged out of memory
    • swap - swap usage, in bytes
    • active_anon - anonymous and swap cache on active least-recently-used (LRU) list, including tmpfs (shmem), in bytes
    • inactive_anon - anonymous and swap cache on inactive LRU list, including tmpfs (shmem), in bytes
    • active_file - file-backed memory on active LRU list, in bytes
    • inactive_file - file-backed memory on inactive LRU list, in bytes
    • unevictable - memory that cannot be reclaimed, in bytes
    • hierarchical_memory_limit - memory limit for the hierarchy that contains the memory cgroup, in bytes
    • hierarchical_memsw_limit - memory plus swap limit for the hierarchy that contains the memory cgroup, in bytes

8. net_cls

  • net_cls.classid - 指定tc的handle，经过tc实现网络控制

9.net_prio 指定task网络设备优先级

  • net_prio.prioidx - a read-only file which contains a unique integer value that the kernel uses as an internal representation of this cgroup.
  • net_prio.ifpriomap - 网络设备使用优先级 - <network_interface> <priority>

10.其余

  • tasks - 该cgroup的全部进程pid
  • cgroup.event_control - event api
  • cgroup.procs - thread group id
  • release_agent(present in the root cgroup only) - 根据notify_on_release是否在task为空时执行的脚本
  • notify_on_release - 当cgroup中没有task时是否执行release_agent

总结

  1. 本文总结了cgroup的操做方法和详细的可配置项，为对更好的控制系统中的资源分配打下基础
  2. 对于限制资源分配的两个场景，在针对特殊APP的场景中可进行很是细致的调优，而在通用的资源隔离的角度上看，可能更关注的是CPU和内存相关的主要属性
相关文章
相关标签/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公众号
   欢迎关注本站公众号,获取更多信息
联系我们 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程