How to use bcc-tools for dynamic kernel tracing

In Red Hat Enterprise Linux 8.1, Red Hat ships a set of fully supported on x86_64 dynamic kernel tracing tools, called bcc-tools, that make use of a kernel technology called extended Berkeley Packet Filter (eBPF). With these tools, you can quickly gain insight into certain aspects of system performance that would have previously required more time and effort from the system and operator.

The eBPF technology allows dynamic kernel tracing without requiring kernel modules (like systemtap) or rebooting of the kernel (as with debug kernels). eBPF accomplishes this while maintaining minimal overhead for each trace point, making these tools an ideal way to instrument running kernels in production.

To ensure that an eBPF program will not harm the running kernel, tools built on eBPF go through the following process when instantiated by root on the command line:

  • The program is compiled into eBPF bytecode.
  • Loaded into the kernel.
  • Run through a technology called the eBPF verifier to ensure that the program will not harm the running kernel.
  • Upon passing the verifier, it begins execution. If it does not pass the verifier, the code is unloaded and does not execute.

That said, bear in mind that you are still inserting tracing and some system calls are called significantly more than others, so depending on what you are tracing, there may be increased overhead.

If you are interested in more information on eBPF in general, please see Stanislav Kozina’s blog: Introduction to eBPF in Red Hat Enterprise Linux 7.


With RHEL 8.1, bcc-tools has gone fully supported on x86. To install bcc-tools on RHEL 7 (7.6+) and RHEL 8, run yum install as root:

$ uname -r
$  cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.7 (Maipo)
$ yum install bcc-tools
  bcc-tools.x86_64 0:0.10.0-1.el7
Dependency Installed:
  bcc.x86_64 0:0.10.0-1.el7                kernel-devel.x86_64 0:3.10.0-1160.21.1.el7
  llvm-private.x86_64 0:7.0.1-1.el7        python-bcc.x86_64 0:0.10.0-1.el7

$ pwd
$  ls
argdist       drsnoop         memleak         pythonstat   tclobjnew
bashreadline  execsnoop       mountsnoop      reset-trace  tclstat
biolatency    ext4dist        mysqld_qslower  rubycalls    tcpaccept
biosnoop      ext4slower      nfsdist         rubyflow     tcpconnect
biotop        filelife        nfsslower       rubygc       tcpconnlat
bitesize      fileslower      nodegc          rubyobjnew   tcpdrop
bpflist       filetop         nodestat        rubystat     tcplife
btrfsdist     funccount       offcputime      runqlat      tcpretrans
btrfsslower   funclatency     offwaketime     runqlen      tcpsubnet
cachestat     funcslower      oomkill         runqslower   tcptop
cachetop      gethostlatency  opensnoop       shmsnoop     tcptracer
capable       hardirqs        perlcalls       slabratetop  tplist
cobjnew       javacalls       perlflow        sofdsnoop    trace
cpudist       javaflow        perlstat        softirqs     ttysnoop
cpuunclaimed  javagc          phpcalls        solisten     vfscount
dbslower      javaobjnew      phpflow         sslsniff     vfsstat
dbstat        javastat        phpstat         stackcount   wakeuptime
dcsnoop       javathreads     pidpersec       statsnoop    xfsdist
dcstat        killsnoop       profile         syncsnoop    xfsslower
deadlock      lib             pythoncalls     syscount
deadlock.c    llcstat         pythonflow      tclcalls
doc           mdflush         pythongc        tclflow  

bcc-tools Framework

Before we dive into the different types of tools that are included in bcc-tools, it’s important to note a few things:

  • All of these tools live in /usr/share/bcc/tools.
  • These tools must run as the root user as any eBPF program can read kernel data. As such, injecting eBPF bytecode as a regular user is not allowed in RHEL 8.1.
  • Each tool has a man page. To view the man page, run man . These man pages include descriptions of the tools, provide the options that can be called, and have information on the expected overhead of the specific tool.

Since there are a lot of tools in bcc-tools, I’m going to divide the tools into the following classes and then we’ll dive into each class:

  • The Snoops
  • Latency Detectors
  • Slower
  • Top Up with bcc-tools
  • Java/Perl/Python/Ruby