Using cgroups to limit Memory usage
The memory controller isolates the memory behaviour of a group of tasks from the rest of the system.
$ mount | egrep "/cgroup |/memory"
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
$ lssubsys -am | grep memory
memory /sys/fs/cgroup/memory
In this post, we will learn how to use the following control files to limit and monitor memory usage for the user tasks.
- memory.usage_in_bytes - show current usage for memory
- memory.limit_in_bytes - set/show limit of memory usage
Create memory control group
Install libcgroup package to manage cgroups:
$ yum install libcgroup libcgroup-tools
Create memory control group:
$ cgcreate -g memory:/memlimited
$ lscgroup | grep memory
memory:/
memory:/memlimited
$ ls /sys/fs/cgroup/memory/memlimited/
cgroup.clone_children memory.kmem.slabinfo memory.memsw.failcnt memory.soft_limit_in_bytes
cgroup.event_control memory.kmem.tcp.failcnt memory.memsw.limit_in_bytes memory.stat
cgroup.procs memory.kmem.tcp.limit_in_bytes memory.memsw.max_usage_in_bytes memory.swappiness
memory.failcnt memory.kmem.tcp.max_usage_in_bytes memory.memsw.usage_in_bytes memory.usage_in_bytes
memory.force_empty memory.kmem.tcp.usage_in_bytes memory.move_charge_at_immigrate memory.use_hierarchy
memory.kmem.failcnt memory.kmem.usage_in_bytes memory.numa_stat notify_on_release
memory.kmem.limit_in_bytes memory.limit_in_bytes memory.oom_control tasks
memory.kmem.max_usage_in_bytes memory.max_usage_in_bytes memory.pressure_level
Limit the memory usage
Using control files directly
$ echo 32G > /sys/fs/cgroup/memory/memlimited/memory.limit_in_bytes
$ cat /sys/fs/cgroup/memory/memlimited/memory.limit_in_bytes
34359738368
Using libcgroup tools
Limit the memory usage:
$ cgset -r memory.limit_in_bytes=32G memlimited
$ cgget -r memory.limit_in_bytes memlimited
memlimited:
memory.limit_in_bytes: 34359738368
Verify the memory usage
Unlimit the memory usage
$ cgset -r memory.limit_in_bytes=-1 memlimited
$ cgget -r memory.limit_in_bytes memlimited
memlimited:
memory.limit_in_bytes: 9223372036854771712
Use fio to write 50G data:
$ echo 3 > /proc/sys/vm/drop_caches
$ cgexec -g memory:memlimited fio --blocksize=64k --ioengine=libaio --readwrite=write --filesize=50G --group_reporting --direct=0 --iodepth=128 --end_fsync=1 --name=job1 --filename=/mnt/fio.dat
Verify the memory usage is unlimited:
$ while true; do cat /sys/fs/cgroup/memory/memlimited/memory.usage_in_bytes; sleep 5; done
14143488
14143488
819499008
13288558592
25772953600
38258790400
50776608768
55638511616
55638429696
55638478848
55638528000
55638577152
55210229760
55210229760
^C
Limit the memory usage to 32GB
$ cgset -r memory.limit_in_bytes=32G memlimited
$ cgget -r memory.limit_in_bytes memlimited
memlimited:
memory.limit_in_bytes: 34359738368
Use fio to write 50G data:
$ echo 3 > /proc/sys/vm/drop_caches
$ cgexec -g memory:memlimited fio --blocksize=64k --ioengine=libaio --readwrite=write --filesize=50G --group_reporting --direct=0 --iodepth=128 --end_fsync=1 --name=job1 --filename=/mnt/fio.dat
Verify the memory usage is limited to 32GB:
$ while true; do cat /sys/fs/cgroup/memory/memlimited/memory.usage_in_bytes; sleep 5; done
9134080
6819614720
18048208896
29089763328
34359726080
34359672832
34359607296
34359717888
34359635968
34359619584
34280120320
34280120320
^C
From vmstat output, the cache usage is limited to 32GB. There is also swapping out activity due the memory pressure.
$ vmstat 5 -t
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st UTC
0 0 0 1053891776 968 142992 0 0 0 30 0 0 0 0 100 0 0 2022-12-23 22:04:30
2 0 0 1045755200 968 8266836 0 0 1697 0 3789 727 2 1 98 0 0 2022-12-23 22:04:35
1 0 0 1034800320 976 19222628 0 0 0 2 1770 575 0 1 99 0 0 2022-12-23 22:04:40
1 0 0 1024034304 984 29988256 0 0 0 163843 2999 780 0 1 99 0 0 2022-12-23 22:04:45
1 1 0 1020353664 992 33665560 0 0 0 985500 8317 2414 0 1 98 0 0 2022-12-23 22:04:50
1 1 317440 1020354112 1000 33666764 0 63474 3 1518376 18524 2604 0 1 98 1 0 2022-12-23 22:04:55
1 1 340480 1020353472 1008 33666660 0 4571 0 1602014 15377 2381 0 1 98 1 0 2022-12-23 22:05:00
2 0 340480 1020355904 1016 33667212 39 0 39 1632866 22166 70618 0 1 98 0 0 2022-12-23 22:05:05
1 0 340480 1020356032 1020 33667228 0 0 0 1861355 27437 108029 0 1 99 0 0 2022-12-23 22:05:10
2 0 340480 1020356224 1024 33667236 0 0 0 1874438 28732 111364 0 1 98 0 0 2022-12-23 22:05:15
0 0 512 1020432704 1024 33599516 212 0 550 915429 13644 56805 0 1 99 0 0 2022-12-23 22:05:20
0 0 512 1020432960 1028 33599516 0 0 3 4 165 149 0 0 100 0 0 2022-12-23 22:05:25
^C
Limit the memory usage for the tasks in the current bash
$ cat /sys/fs/cgroup/memory/memlimited/tasks
$ echo $$ > /sys/fs/cgroup/memory/memlimited/tasks
$ cat /sys/fs/cgroup/memory/memlimited/tasks
27875
28889
$ ps -ef | egrep "27875|29023" | grep -v grep
root 27875 27873 0 21:11 pts/0 00:00:17 -bash
root 29026 27875 0 22:11 pts/0 00:00:00 ps -ef
Use fio to write 50G data:
$ fio --blocksize=64k --ioengine=libaio --readwrite=write --filesize=50G --group_reporting --direct=0 --iodepth=128 --end_fsync=1 --name=job1 --filename=/mnt/fio.dat
Verify the memory usage is limited to 32GB:
$ while true; do cat /sys/fs/cgroup/memory/memlimited/memory.usage_in_bytes; sleep 5; done
22835200
22835200
22835200
10150678528
21453983744
32693850112
34359607296
34359607296
34359738368
34359730176
34359730176
34359660544
34280062976
^C
$ vmstat 5 -t
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st UTC
0 0 512 1053886784 1004 144860 0 0 0 31 0 0 0 0 100 0 0 2022-12-23 22:09:17
0 0 512 1053886528 1004 144824 0 0 0 0 143 148 0 0 100 0 0 2022-12-23 22:09:22
1 0 512 1050907776 1004 3109688 0 0 1660 3 2690 566 1 0 98 0 0 2022-12-23 22:09:27
1 0 512 1039836288 1012 14180528 0 0 0 2 1699 508 0 1 99 0 0 2022-12-23 22:09:32
1 0 512 1028822720 1020 25194536 0 0 0 2 1625 494 0 1 99 0 0 2022-12-23 22:09:37
1 1 512 1020348800 1028 33665604 0 0 0 385026 4478 1313 0 1 99 0 0 2022-12-23 22:09:42
1 1 46080 1020340224 1036 33676796 0 9090 0 1563114 13781 2301 0 1 98 1 0 2022-12-23 22:09:47
1 1 340480 1020338688 1044 33676544 0 58904 0 1557223 19212 2580 0 1 98 1 0 2022-12-23 22:09:52
1 1 340480 1020337792 1052 33676376 33 0 33 1481028 14133 5776 0 1 98 1 0 2022-12-23 22:09:57
1 0 340480 1020342208 1056 33676396 0 0 184 1877767 27447 114955 0 1 98 0 0 2022-12-23 22:10:02
1 1 340480 1020341056 1056 33677100 0 0 0 1817494 27635 101675 0 1 98 0 0 2022-12-23 22:10:07
1 0 195144 1020350208 1060 33676988 83 0 83 1872111 27068 113280 0 1 98 0 0 2022-12-23 22:10:12
^C