Performance

General steps for system performance analysis

This post tries to provide general steps for system performance analysis. We are going to cover the following objectives. * The performance goals * Workload characterization * Drill-down analysis * Traditional performance tools * BCC/BPF performance tools The Goals of performance analysis In general, the goals of performance analysis are to improve end-user performance

Linux performance analysis in 60 seconds

Performance tools checklist The following tools can be used for the first look at any performance issues. By executing these commands in the poorly performing Linux system, it would help solve the performance issue or reveal the clues for the next step of performance issue analysis. * uptime * dmesg | tail * lscpu

Analyze Cockroach performance from the built-in DB console

In this post, we try to understand how to analyze CockroachDB performance by monitoring its built-in DB console.

Flame Graph and stack trace visualization

A stack trace is series of functions which show the code path. With the Linux perf(1) tool, the stack samples are summarized as call tree. With the BCC profile(8) tool, the unique stack trace shows as a count. Both tools are very useful when there are only a

Why performance tools need BPF

BPF is efficient, production safe and built into the Linux kernel. With BPF, you can use the performance tools in production environment without needing to add any new kernel components. The following example summarizes the block I/O latency as a histogram. $ biolatency Tracing block device I/O... Hit Ctrl-C

Introduction to BPF performance tools

BPF stands for Berkeley Packet Filter, which was firstly developed to improve the performance of packet capture tools. It was rewritten later and turned into a general-purpose execution engine including the creation of performance analysis tools.

Getting started with bpftrace

bpftrace is a high-level tracing language for Linux enhanced Berkeley Packet Filter (eBPF) available in recent Linux kernels (4.x).

Getting started with BCC (BPF Compiler Collection)

BCC is a toolkit for creating efficient kernel tracing and manipulation programs, and includes several useful tools and examples.

Using systemtap to analyze latency of the kernel module function

In this post, we continue to explore how to use SystemTap to analyze the latency of the kernel module function. In the following example, we want to analyze the latency of the function "nfsd_vfs_write" from the kernel module "nfsd". Deploy the SystemTap packages Refer to this post to

Getting started with systemtap for Linux system profiling

Intro to SystemTap SystemTap is a tracing and probing tool that allows users to study and monitor the activities of the operating system (particularly, the kernel) in fine detail. It provides information similar to the output of tools like netstat, ps, top, and iostat; however, SystemTap is designed to provide

Using ftrace to analyze latency of the kernel module function

In this example, we study the latency of the function "nfsd_vfs_write" from kernel module "nfsd". ftrace configuration The following ftrace options are used in this example. There are 8 nfsd processes to be traced. * current_tracer: function_graph * pids: 3591 3592 3593 3594 3595 3596 3597 3598 * filters:

Dynamically tracing with user-defined tracepoint in perf

In this post, we are going to explore how to use perf for dynamic tracing with user-defined tracepoint. When we say dynamic tracing, the kernel event(function) to be traced is not predefined in perf. Instead, we add the tracepoint event manually. We will study it based on a real

Flame graph - A visualization for profiling and debugging

Flame graphs can be generated from the output of many different software profilers, including profiles for different resources and event types. The article written by Brendan Gregg describes how the flame graph works. We can start to work with the flame graph by following commands. $ yum install git $ git clone

What is Performance Engineering

Useful tools in Linux operating system

Performance tools * vmstat * iostat * ifstat * netstat * nfsstat * mpstat * nstat * dstat * sar * iftop * pidstat * xosview Benchmark tools * fio * iozone * iperf * netperf * vdbench * sysbench * pgbench * YCSB * SPEC SFS 2014/SPECstorage Solution 2020 * VMmark Debugging tools * htop * lslk * lsof * top Process tracing * ltrace * strace * pstack/gstack * ftrace * systemtap * ps * pmap * blktrace * ebpf Binary

Must-read books and articles for Software and Performance Engineering

Books * The Art of Computer Programming * Understanding the Linux Kernel, 3rd Edition * Linux Device Drivers, 3rd Edition * Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems, 1st Edition * Building Microservices: Designing Fine-Grained Systems 1st Edition * Site Reliability Engineering: How Google Runs Production Systems 1st E

Using bcc-tools for dynamic kernel tracing

In Red Hat Enterprise Linux 8.1, Red Hat ships a set of fully supported on x86_64 dynamic kernel tracing tools, called bcc-tools, that make use of a kernel technology called extended Berkeley Packet Filter (eBPF). With these tools, you can quickly gain insight into certain aspects of system

blktrace - A block layer IO tracing utility

What can blktrace do? Let's dive into the block device layer and see how the I/O are handled in disk queues. The following stack shows the I/O paths including the block device layer. The application can issue I/O requests directly to block device or through file systems.

Understanding await in iostat

What's meaning of await in iostat? The following is the description provided for await field in iostat man page. $ man iostat await The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the

Buffered and Direct IO

Buffered and Direct I/O The VxFS responds with read-ahead for sequential read I/O. This results in buffered I/O. The data is prefetched and retained in buffers, in anticipation of application asking for it. The data buffers are commonly referred to as the VxFS buffer cache. This is

Kubernetes and Gluster performance

Kubernetes and Gluster Intro Kubernetes, also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications. Gluster is a scalable, distributed file system that aggregates disk storage resources from multiple servers into a single global namespace. Gluster performance study In this article, we will

Perf - The official Linux profiler

Introduction perf, aka perf_events, is the official Linux profiler and included in the Linux kernel source under tools/perf. It can instrument CPU performance counters, tracepoints, kprobes, and uprobes. It is capable of system profiling. Performance Monitoring Counter(PMC) Show system wide PMC statistics The following example shows PMC

Analyze the library and system calls

When the application source code is not available, we can trace the function calls during its runtime in order to understand how the application code works. Library calls take place in user space. System calls is the fundamental interface between application and the kernel. System calls are not invoked directly

Memory fragmentation

Memory fragmentation Memory page availability can be checked from /proc/buddyinfo as below. $ cat /proc/buddyinfo Node 0, zone DMA 1 0 1 0 1 1 1 0 1 1 3 Node 0, zone DMA32 3342 2441 2138 5025 1871 236 3 0 0 0 0 Node 0, zone Normal

What is patrol read

What is patrolread The Patrol Read feature is designed as a preventative measure to ensure hard drive health and data integrity. Patrol Read scans for and resolves potential problems on configured hard drives. Patrol Read Behavior The following is an overview of Patrol Read behavior: * Patrol Read runs on all