How to use bcc-tools for dynamic kernel tracing
In Red Hat Enterprise Linux 8.1, Red Hat ships a set of fully supported on x86_64 dynamic kernel tracing tools, called bcc-tools, that make use of a kernel technology called extended Berkeley Packet Filter (eBPF). With these tools, you can quickly gain insight into certain aspects of system performance that would have previously required more time and effort from the system and operator.
The eBPF technology allows dynamic kernel tracing without requiring kernel modules (like systemtap) or rebooting of the kernel (as with debug kernels). eBPF accomplishes this while maintaining minimal overhead for each trace point, making these tools an ideal way to instrument running kernels in production.
To ensure that an eBPF program will not harm the running kernel, tools built on eBPF go through the following process when instantiated by root on the command line:
- The program is compiled into eBPF bytecode.
- Loaded into the kernel.
- Run through a technology called the eBPF verifier to ensure that the program will not harm the running kernel.
- Upon passing the verifier, it begins execution. If it does not pass the verifier, the code is unloaded and does not execute.
That said, bear in mind that you are still inserting tracing and some system calls are called significantly more than others, so depending on what you are tracing, there may be increased overhead.
If you are interested in more information on eBPF in general, please see Stanislav Kozina’s blog: Introduction to eBPF in Red Hat Enterprise Linux 7.
Installation
With RHEL 8.1, bcc-tools has gone fully supported on x86. To install bcc-tools on RHEL 7 (7.6+) and RHEL 8, run yum install as root:
$ uname -r
3.10.0-1062.40.1.el7.x86_64
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.7 (Maipo)
$ yum install bcc-tools
Installed:
bcc-tools.x86_64 0:0.10.0-1.el7
Dependency Installed:
bcc.x86_64 0:0.10.0-1.el7 kernel-devel.x86_64 0:3.10.0-1160.21.1.el7
llvm-private.x86_64 0:7.0.1-1.el7 python-bcc.x86_64 0:0.10.0-1.el7
$ pwd
/usr/share/bcc/tools
$ ls
argdist drsnoop memleak pythonstat tclobjnew
bashreadline execsnoop mountsnoop reset-trace tclstat
biolatency ext4dist mysqld_qslower rubycalls tcpaccept
biosnoop ext4slower nfsdist rubyflow tcpconnect
biotop filelife nfsslower rubygc tcpconnlat
bitesize fileslower nodegc rubyobjnew tcpdrop
bpflist filetop nodestat rubystat tcplife
btrfsdist funccount offcputime runqlat tcpretrans
btrfsslower funclatency offwaketime runqlen tcpsubnet
cachestat funcslower oomkill runqslower tcptop
cachetop gethostlatency opensnoop shmsnoop tcptracer
capable hardirqs perlcalls slabratetop tplist
cobjnew javacalls perlflow sofdsnoop trace
cpudist javaflow perlstat softirqs ttysnoop
cpuunclaimed javagc phpcalls solisten vfscount
dbslower javaobjnew phpflow sslsniff vfsstat
dbstat javastat phpstat stackcount wakeuptime
dcsnoop javathreads pidpersec statsnoop xfsdist
dcstat killsnoop profile syncsnoop xfsslower
deadlock lib pythoncalls syscount
deadlock.c llcstat pythonflow tclcalls
doc mdflush pythongc tclflow
bcc-tools Framework
Before we dive into the different types of tools that are included in bcc-tools, it’s important to note a few things:
- All of these tools live in /usr/share/bcc/tools.
- These tools must run as the root user as any eBPF program can read kernel data. As such, injecting eBPF bytecode as a regular user is not allowed in RHEL 8.1.
- Each tool has a man page. To view the man page, run man . These man pages include descriptions of the tools, provide the options that can be called, and have information on the expected overhead of the specific tool.
Since there are a lot of tools in bcc-tools, I’m going to divide the tools into the following classes and then we’ll dive into each class:
- The Snoops
- Latency Detectors
- Slower
- Top Up with bcc-tools
- Java/Perl/Python/Ruby