How to use sar to understand Linux system activity
About sadc and sar
sadc is known as system activity data collector. It samples system data a specified number of times (count) at a specified interval measured in seconds (interval). It writes in binary format to the specified outfile or to standard output. The sadc command is intended to be used as a backend to the sar command. The sar command can be used to collect, report or save system activity information.
The following are examples for some of the often used options to understand system activity including CPU, Memory, Disk IO and network.
Report CPU utilization
The ALL keyword with “-u” indicates that all the CPU fields should be displayed.
1 | [root@host1 ~]# sar -u 1 2 |
Report memory statistics
[root@host1 ~]# sar -R 1 2
Linux 5.7.12-1.el7.elrepo.x86_64 (host1) 07/06/2022 _x86_64_ (96 CPU)
07:00:19 PM frmpg/s bufpg/s campg/s
07:00:20 PM -10.00 0.00 0.00
07:00:21 PM 8.00 0.00 0.00
Average: -1.00 0.00 0.00
Report memory utilization statistics
[root@host1 ~]# sar -r 1 2
Linux 5.7.12-1.el7.elrepo.x86_64 (host1) 07/06/2022 _x86_64_ (96 CPU)
06:47:59 PM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
06:48:00 PM 1052993760 3499308 0.33 155084 226796 1805324 0.17 343248 178520 184
06:48:01 PM 1052993776 3499292 0.33 155084 226796 1805324 0.17 343464 178520 184
Average: 1052993768 3499300 0.33 155084 226796 1805324 0.17 343356 178520 184
Report swap space utilization statistics
[root@host1 ~]# sar -S 1 2
Linux 5.7.12-1.el7.elrepo.x86_64 (host1) 07/06/2022 _x86_64_ (96 CPU)
06:47:25 PM kbswpfree kbswpused %swpused kbswpcad %swpcad
06:47:26 PM 4194300 0 0.00 0 0.00
06:47:27 PM 4194300 0 0.00 0 0.00
Average: 4194300 0 0.00 0 0.00
Report swapping statistics
[root@host1 ~]# sar -W 1 2
Linux 5.7.12-1.el7.elrepo.x86_64 (host1) 07/06/2022 _x86_64_ (96 CPU)
06:50:10 PM pswpin/s pswpout/s
06:50:11 PM 0.00 0.00
06:50:12 PM 0.00 0.00
Average: 0.00 0.00
Report paging statistics
[root@host1 ~]# sar -B 1 2
Linux 5.7.12-1.el7.elrepo.x86_64 (host1) 07/06/2022 _x86_64_ (96 CPU)
06:51:29 PM pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff
06:51:30 PM 0.00 0.00 22.00 0.00 48.00 0.00 0.00 0.00 0.00
06:51:31 PM 0.00 0.00 19.00 0.00 47.00 0.00 0.00 0.00 0.00
Average: 0.00 0.00 20.50 0.00 47.50 0.00 0.00 0.00 0.00
Report hugepages utilization statistics
[root@host1 ~]# sar -H 1 2
Linux 5.7.12-1.el7.elrepo.x86_64 (host1) 07/06/2022 _x86_64_ (96 CPU)
06:54:34 PM kbhugfree kbhugused %hugused
06:54:35 PM 0 0 0.00
06:54:36 PM 0 0 0.00
Average: 0 0 0.00
Report task creation and system switching activity
[root@host1 ~]# sar -w 1 2
Linux 5.7.12-1.el7.elrepo.x86_64 (host1) 07/06/2022 _x86_64_ (96 CPU)
06:50:34 PM proc/s cswch/s
06:50:35 PM 0.00 524.00
06:50:36 PM 0.00 721.00
Average: 0.00 622.50
Report I/O and transfer rate statistics
[root@host1 ~]# sar -b 1 2
Linux 5.7.12-1.el7.elrepo.x86_64 (host1) 07/06/2022 _x86_64_ (96 CPU)
06:52:14 PM tps rtps wtps bread/s bwrtn/s
06:52:15 PM 0.00 0.00 0.00 0.00 0.00
06:52:16 PM 0.00 0.00 0.00 0.00 0.00
Average: 0.00 0.00 0.00 0.00 0.00
Report activity for each block device
[root@host1 ~]# sar -d 1 2
Linux 5.7.12-1.el7.elrepo.x86_64 (host1) 07/06/2022 _x86_64_ (96 CPU)
06:53:04 PM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
06:53:05 PM dev259-0 1.00 0.00 8.00 8.00 0.00 0.00 1.00 0.10
06:53:05 PM dev259-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:53:05 PM dev259-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:53:05 PM dev259-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:53:05 PM dev259-7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:53:05 PM dev259-8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:53:05 PM dev259-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:53:05 PM dev259-10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:53:05 PM dev259-11 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:53:05 PM dev259-12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:53:05 PM dev253-0 1.00 0.00 8.00 8.00 0.00 0.00 1.00 0.10
06:53:05 PM dev253-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:53:05 PM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
06:53:06 PM dev259-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:53:06 PM dev259-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:53:06 PM dev259-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:53:06 PM dev259-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:53:06 PM dev259-7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:53:06 PM dev259-8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:53:06 PM dev259-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:53:06 PM dev259-10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:53:06 PM dev259-11 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:53:06 PM dev259-12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:53:06 PM dev253-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:53:06 PM dev253-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
Average: dev259-0 0.50 0.00 4.00 8.00 0.00 0.00 1.00 0.05
Average: dev259-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: dev259-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: dev259-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: dev259-7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: dev259-8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: dev259-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: dev259-10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: dev259-11 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: dev259-12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: dev253-0 0.50 0.00 4.00 8.00 0.00 0.00 1.00 0.05
Average: dev253-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Report network statistics
[root@host1 ~]# sar -n DEV 1 2
Linux 5.7.12-1.el7.elrepo.x86_64 (host1) 07/06/2022 _x86_64_ (96 CPU)
06:56:34 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
06:56:35 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:56:35 PM eth0 15.00 1.00 0.89 0.14 0.00 0.00 0.00
06:56:35 PM eth3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:56:35 PM eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:56:35 PM eth2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:56:35 PM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:56:35 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
06:56:36 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:56:36 PM eth0 16.00 1.00 0.94 0.74 0.00 0.00 0.00
06:56:36 PM eth3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:56:36 PM eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:56:36 PM eth2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:56:36 PM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
Average: lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: eth0 15.50 1.00 0.92 0.44 0.00 0.00 0.00
Average: eth3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: eth2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Save and extract records from file
Save the readings in the file in binary form. Each reading is in a separate record. The default value of the filename parameter is the current daily data file, the /var/log/sa/sadd file. The -o option is exclusive of the -f option. All the data available from the kernel are saved in the file (in fact, sar calls its data collector sadc with the option “-S ALL”).
[root@host1 ~]# sar -o sar.out.1.2
[root@host1 ~]# sar -f sar.out.1.2 -u
Linux 5.7.12-1.el7.elrepo.x86_64 (host1) 07/06/2022 _x86_64_ (96 CPU)
07:04:45 PM CPU %user %nice %system %iowait %steal %idle
07:04:46 PM all 0.02 0.00 0.05 0.00 0.00 99.93
07:04:47 PM all 0.06 0.00 0.11 0.00 0.00 99.82
Average: all 0.04 0.00 0.08 0.00 0.00 99.87