In Red Hat Enterprise Linux 8.1, Red Hat ships a set of fully supported on x86_64 dynamic kernel tracing tools, called bcc-tools, that make use of a kernel technology called extended Berkeley Packet Filter (eBPF). With these tools, you can quickly gain insight into certain aspects of system performance that would have previously required more time and effort from the system and operator.
The eBPF technology allows dynamic kernel tracing without requiring kernel modules (like systemtap) or rebooting of the kernel (as with debug kernels). eBPF accomplishes this while maintaining minimal overhead for each trace point, making these tools an ideal way to instrument running kernels in production.
To ensure that an eBPF program will not harm the running kernel, tools built on eBPF go through the following process when instantiated by root on the command line:
- The program is compiled into eBPF bytecode.
- Loaded into the kernel.
- Run through a technology called the eBPF verifier to ensure that the program will not harm the running kernel.
- Upon passing the verifier, it begins execution. If it does not pass the verifier, the code is unloaded and does not execute.
That said, bear in mind that you are still inserting tracing and some system calls are called significantly more than others, so depending on what you are tracing, there may be increased overhead.
If you are interested in more information on eBPF in general, please see Stanislav Kozina’s blog: Introduction to eBPF in Red Hat Enterprise Linux 7.
With RHEL 8.1, bcc-tools has gone fully supported on x86. To install bcc-tools on RHEL 7 (7.6+) and RHEL 8, run yum install as root:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 $ uname -r 3.10.0-1062.40.1.el7.x86_64 $ cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.7 (Maipo) $ yum install bcc-tools Installed: bcc-tools.x86_64 0:0.10.0-1.el7 Dependency Installed: bcc.x86_64 0:0.10.0-1.el7 kernel-devel.x86_64 0:3.10.0-1160.21.1.el7 llvm-private.x86_64 0:7.0.1-1.el7 python-bcc.x86_64 0:0.10.0-1.el7 $ pwd /usr/share/bcc/tools $ ls argdist drsnoop memleak pythonstat tclobjnew bashreadline execsnoop mountsnoop reset-trace tclstat biolatency ext4dist mysqld_qslower rubycalls tcpaccept biosnoop ext4slower nfsdist rubyflow tcpconnect biotop filelife nfsslower rubygc tcpconnlat bitesize fileslower nodegc rubyobjnew tcpdrop bpflist filetop nodestat rubystat tcplife btrfsdist funccount offcputime runqlat tcpretrans btrfsslower funclatency offwaketime runqlen tcpsubnet cachestat funcslower oomkill runqslower tcptop cachetop gethostlatency opensnoop shmsnoop tcptracer capable hardirqs perlcalls slabratetop tplist cobjnew javacalls perlflow sofdsnoop trace cpudist javaflow perlstat softirqs ttysnoop cpuunclaimed javagc phpcalls solisten vfscount dbslower javaobjnew phpflow sslsniff vfsstat dbstat javastat phpstat stackcount wakeuptime dcsnoop javathreads pidpersec statsnoop xfsdist dcstat killsnoop profile syncsnoop xfsslower deadlock lib pythoncalls syscount deadlock.c llcstat pythonflow tclcalls doc mdflush pythongc tclflow
Before we dive into the different types of tools that are included in bcc-tools, it’s important to note a few things:
- All of these tools live in /usr/share/bcc/tools.
- These tools must run as the root user as any eBPF program can read kernel data. As such, injecting eBPF bytecode as a regular user is not allowed in RHEL 8.1.
- Each tool has a man page. To view the man page, run man
. These man pages include descriptions of the tools, provide the options that can be called, and have information on the expected overhead of the specific tool.
Since there are a lot of tools in bcc-tools, I’m going to divide the tools into the following classes and then we’ll dive into each class:
- The Snoops
- Latency Detectors
- Top Up with bcc-tools