Using cgroups to limit Memory usage

Using cgroups to limit Memory usage
Photo by Markus Winkler / Unsplash

The memory controller isolates the memory behaviour of a group of tasks from the rest of the system.

$ mount | egrep "/cgroup |/memory"
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)

$ lssubsys -am | grep memory
memory /sys/fs/cgroup/memory

In this post, we will learn how to use the following control files to limit and monitor memory usage for the user tasks.

  • memory.usage_in_bytes - show current usage for memory
  • memory.limit_in_bytes - set/show limit of memory usage

Create memory control group

Install libcgroup package to manage cgroups:

$ yum install libcgroup libcgroup-tools

Create memory control group:

$ cgcreate -g memory:/memlimited
$ lscgroup | grep memory
memory:/
memory:/memlimited

$ ls /sys/fs/cgroup/memory/memlimited/
cgroup.clone_children           memory.kmem.slabinfo                memory.memsw.failcnt             memory.soft_limit_in_bytes
cgroup.event_control            memory.kmem.tcp.failcnt             memory.memsw.limit_in_bytes      memory.stat
cgroup.procs                    memory.kmem.tcp.limit_in_bytes      memory.memsw.max_usage_in_bytes  memory.swappiness
memory.failcnt                  memory.kmem.tcp.max_usage_in_bytes  memory.memsw.usage_in_bytes      memory.usage_in_bytes
memory.force_empty              memory.kmem.tcp.usage_in_bytes      memory.move_charge_at_immigrate  memory.use_hierarchy
memory.kmem.failcnt             memory.kmem.usage_in_bytes          memory.numa_stat                 notify_on_release
memory.kmem.limit_in_bytes      memory.limit_in_bytes               memory.oom_control               tasks
memory.kmem.max_usage_in_bytes  memory.max_usage_in_bytes           memory.pressure_level

Limit the memory usage

Using control files directly

$ echo 32G > /sys/fs/cgroup/memory/memlimited/memory.limit_in_bytes
$ cat /sys/fs/cgroup/memory/memlimited/memory.limit_in_bytes
34359738368

Using libcgroup tools

Limit the memory usage:

$ cgset -r memory.limit_in_bytes=32G memlimited
$ cgget -r memory.limit_in_bytes memlimited
memlimited:
memory.limit_in_bytes: 34359738368

Verify the memory usage

Unlimit the memory usage

$ cgset -r memory.limit_in_bytes=-1 memlimited
$ cgget -r memory.limit_in_bytes memlimited
memlimited:
memory.limit_in_bytes: 9223372036854771712

Use fio to write 50G data:

$ echo 3 > /proc/sys/vm/drop_caches
$ cgexec -g memory:memlimited fio --blocksize=64k --ioengine=libaio --readwrite=write --filesize=50G --group_reporting --direct=0 --iodepth=128 --end_fsync=1 --name=job1 --filename=/mnt/fio.dat

Verify the memory usage is unlimited:

$ while true; do cat /sys/fs/cgroup/memory/memlimited/memory.usage_in_bytes; sleep 5; done
14143488
14143488
819499008
13288558592
25772953600
38258790400
50776608768
55638511616
55638429696
55638478848
55638528000
55638577152
55210229760
55210229760
^C

Limit the memory usage to 32GB

$ cgset -r memory.limit_in_bytes=32G memlimited
$ cgget -r memory.limit_in_bytes memlimited
memlimited:
memory.limit_in_bytes: 34359738368

Use fio to write 50G data:

$ echo 3 > /proc/sys/vm/drop_caches
$ cgexec -g memory:memlimited fio --blocksize=64k --ioengine=libaio --readwrite=write --filesize=50G --group_reporting --direct=0 --iodepth=128 --end_fsync=1 --name=job1 --filename=/mnt/fio.dat

Verify the memory usage is limited to 32GB:

$ while true; do cat /sys/fs/cgroup/memory/memlimited/memory.usage_in_bytes; sleep 5; done
9134080
6819614720
18048208896
29089763328
34359726080
34359672832
34359607296
34359717888
34359635968
34359619584
34280120320
34280120320
^C

From vmstat output, the cache usage is limited to 32GB. There is also swapping out activity due the memory pressure.

$ vmstat 5 -t
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- -----timestamp-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st                 UTC
 0  0      0 1053891776    968 142992    0    0     0    30    0    0  0  0 100  0  0 2022-12-23 22:04:30
 2  0      0 1045755200    968 8266836    0    0  1697     0 3789  727  2  1 98  0  0 2022-12-23 22:04:35
 1  0      0 1034800320    976 19222628    0    0     0     2 1770  575  0  1 99  0  0 2022-12-23 22:04:40
 1  0      0 1024034304    984 29988256    0    0     0 163843 2999  780  0  1 99  0  0 2022-12-23 22:04:45
 1  1      0 1020353664    992 33665560    0    0     0 985500 8317 2414  0  1 98  0  0 2022-12-23 22:04:50
 1  1 317440 1020354112   1000 33666764    0 63474     3 1518376 18524 2604  0  1 98  1  0 2022-12-23 22:04:55
 1  1 340480 1020353472   1008 33666660    0 4571     0 1602014 15377 2381  0  1 98  1  0 2022-12-23 22:05:00
 2  0 340480 1020355904   1016 33667212   39    0    39 1632866 22166 70618  0  1 98  0  0 2022-12-23 22:05:05
 1  0 340480 1020356032   1020 33667228    0    0     0 1861355 27437 108029  0  1 99  0  0 2022-12-23 22:05:10
 2  0 340480 1020356224   1024 33667236    0    0     0 1874438 28732 111364  0  1 98  0  0 2022-12-23 22:05:15
 0  0    512 1020432704   1024 33599516  212    0   550 915429 13644 56805  0  1 99  0  0 2022-12-23 22:05:20
 0  0    512 1020432960   1028 33599516    0    0     3     4  165  149  0  0 100  0  0 2022-12-23 22:05:25
^C

Limit the memory usage for the tasks in the current bash

$ cat /sys/fs/cgroup/memory/memlimited/tasks
$ echo $$ > /sys/fs/cgroup/memory/memlimited/tasks
$ cat /sys/fs/cgroup/memory/memlimited/tasks
27875
28889

$ ps -ef | egrep "27875|29023" | grep -v grep
root     27875 27873  0 21:11 pts/0    00:00:17 -bash
root     29026 27875  0 22:11 pts/0    00:00:00 ps -ef

Use fio to write 50G data:

$ fio --blocksize=64k --ioengine=libaio --readwrite=write --filesize=50G --group_reporting --direct=0 --iodepth=128 --end_fsync=1 --name=job1 --filename=/mnt/fio.dat

Verify the memory usage is limited to 32GB:

$ while true; do cat /sys/fs/cgroup/memory/memlimited/memory.usage_in_bytes; sleep 5; done
22835200
22835200
22835200
10150678528
21453983744
32693850112
34359607296
34359607296
34359738368
34359730176
34359730176
34359660544
34280062976
^C
$ vmstat 5 -t
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- -----timestamp-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st                 UTC
 0  0    512 1053886784   1004 144860    0    0     0    31    0    0  0  0 100  0  0 2022-12-23 22:09:17
 0  0    512 1053886528   1004 144824    0    0     0     0  143  148  0  0 100  0  0 2022-12-23 22:09:22
 1  0    512 1050907776   1004 3109688    0    0  1660     3 2690  566  1  0 98  0  0 2022-12-23 22:09:27
 1  0    512 1039836288   1012 14180528    0    0     0     2 1699  508  0  1 99  0  0 2022-12-23 22:09:32
 1  0    512 1028822720   1020 25194536    0    0     0     2 1625  494  0  1 99  0  0 2022-12-23 22:09:37
 1  1    512 1020348800   1028 33665604    0    0     0 385026 4478 1313  0  1 99  0  0 2022-12-23 22:09:42
 1  1  46080 1020340224   1036 33676796    0 9090     0 1563114 13781 2301  0  1 98  1  0 2022-12-23 22:09:47
 1  1 340480 1020338688   1044 33676544    0 58904     0 1557223 19212 2580  0  1 98  1  0 2022-12-23 22:09:52
 1  1 340480 1020337792   1052 33676376   33    0    33 1481028 14133 5776  0  1 98  1  0 2022-12-23 22:09:57
 1  0 340480 1020342208   1056 33676396    0    0   184 1877767 27447 114955  0  1 98  0  0 2022-12-23 22:10:02
 1  1 340480 1020341056   1056 33677100    0    0     0 1817494 27635 101675  0  1 98  0  0 2022-12-23 22:10:07
 1  0 195144 1020350208   1060 33676988   83    0    83 1872111 27068 113280  0  1 98  0  0 2022-12-23 22:10:12
^C

Reference