Surprised! How can the write IOPS be 10x faster than the cloud native disk performance

In a competitive fio write performance benchmark between 3rd party storage solution and cloud native disk, we noticed that the write performance for the 3rd party storage solution is 10x faster than the cloud native. This usually seems impossible since we would argue that no one can beat the raw

fio direct I/O error with 1k blocksize

When to run fio write with small blocksize(e.g. 1k), the following error is seen. $ fio --blocksize=1k --ioengine=libaio --readwrite=randwrite --filesize=2G --group_reporting --direct=1 --iodepth=128 --randrepeat=1 --end_fsync=1 -- name=job1 --numjobs=1 --filename=/mnt/fiomnt/fio.dat job1: (g=0): rw=

Tracking stock financial metrics in tradingview

In this post, we learn how to use tradingview.com to track financial metrics of the interested stock. It helps understand how the financial metrics support the stock price under the hood.

Using ANOVA in R to analyze US COVID data to understand age impact to death rate

A great example to use ANOVA in R to analyze complicated data.

Using R to analyze US COVID pandemic waves and peaks

Goal According to weekly data between 2020 to 2022, we want to get to know the waves and peaks of COVID pandemic in these years. Download the data We will continue to use NCHS(National Center for Health Statistics) as our data source. Visit https://data.cdc.gov/browse?category=

Using R to analyze the quarterly US COVID data

An R example to analyze big amount of data

Getting started with R

Installation The simplest way is to use Homebrew: $ brew install r Another way is to download installation package from https://cloud.r-project.org/ "Hello world" of R Run it from R console Command R will start a R console, and you can run R code inside R console. (base) ➜ benchling

How to redirect the default "404 page not found" error page in Ghost blog

When users access a non-existent page, they may see a page which shows the message "404 page not found" in Ghost blog. We can redirect the page to another page by creating a error-404.hbs file under the theme folder. In the following example, we redirect the users to the

How to install GUI desktop on CentOS 7

Install GNOME desktop via the groups option of yum: $ yum update $ yum -y groups install "GNOME Desktop" Inform the startx command which desktop env to run: $ echo "exec gnome-session" >> ~/.xinitrc Manually start the GUI desktop: $ startx Automaticlly start the GUI desktop after reboot: $ systemctl set-default graphical.target $ reboot

Using signals with kill command in Linux

Usage of signals with kill $ kill -l 1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP 6) SIGABRT 7) SIGBUS 8) SIGFPE 9) SIGKILL 10) SIGUSR1 11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM 16) SIGSTKFLT 17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP 21) SIGTTIN 22)

Using virtctl to access virtual machine in Kubernetes

Install the virtctl client tool Basic VirtualMachineInstance operations can be performed with the stock kubectl utility. However, the virtctl binary utility is required to use advanced features such as: * Serial and graphical console access It also provides convenience commands for: * Starting and stopping VirtualMachineInstances * Live migrating VirtualMachineInstances * Uploading virtual machine

Using node selector to assign virtual machines to a node

In some circumstances, we want to control which node the virtual machine or pod deploys to. The node selector can be used to assign virtual machine or pod to a node. Add label to a node The label can be added to a node from either command line or openshift

Deploy application with kubernetes statefulset

StatefulSet is the workload API object used to manage stateful applications. Like a Deployment, a StatefulSet manages Pods that are based on an identical container spec. Unlike a Deployment, a StatefulSet maintains a sticky identity for each of their Pods. These pods are created from the same spec, but are

Keep a docker container running and not exiting

In the kubernetes environment, we can keep a container(pod) alive and avoid it exits immediately after starting. Method one 1. Use a long-time-run command in Dockerfile CMD ["sh", "-c", "tail -f /dev/null"] or CMD ["sh", "-c", "sleep infinity"] 2. Build the docker image and push to docker repository

Hosting the Ghost blog on GitHub

Install Ghost locally GitHub Pages is a great solution for hosting static website. If you use the Ghost to manage your website, you can install Ghost locally and convert it to a static website in order to host it on Github. Install Ubuntu virtual machine on Windows(optional) Install Ghost-CLI

Pull an image from a private docker registry in Kubernetes Pod

Log in to Docker Hub In order to pull a image from Docker Hub, you must authenticate with a registry. Use docker tool to log in to the Docker Hub as below. A username and password is needed to log in. $ docker login The login process creates or updates the

Deploy kubernetes statefulset with affinity

Here is an example to demonstrate how to assign statefulset and pod to the target worker node. affinity helps define which node will be assigned for the pod by using the node labels. $ kubectl get nodes -o wide --show-labels $ kubectl label nodes portworx.io/fio=true $ cat statefulset-node-select.yaml apiVersion:

Using kubeconfig to configure access to remote Kubernetes cluster

Install kubectl binary The Kubernetes command-line tool, kubectl, allows you to run commands against Kubernetes clusters. You can use kubectl to deploy applications, inspect and manage cluster resources, and view logs. [root@localhost ~]# curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)

Lambda architecture in big data system

What is Lambda Architecture When working with very large data sets, it can take a long time to run the sort of queries that clients need. These queries can't be performed in real time, and often require algorithms such as MapReduce that operate in parallel across the entire data set.

Use sysbench for CockroachDB performance benchmarking

Intro Cockroach uses TPC-C as the official OLTP workload benchmark since it's a more realistic measurement by modeling the real world applications. However, sysbench is a straight-forward throughput/latency benchmarking tool. It is a scriptable multi-threaded benchmark tool based on LuaJIT. It is most frequently used for database benchmarks, but

Using cgroups to limit block device bandwidth

The block I/O controller specifies upper IO rate limits on devices. $ mount | egrep "/cgroup |/blkio" tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) $ lscgroup | grep blkio blkio:/ In this post,

Using cgroups to limit Memory usage

The memory controller isolates the memory behaviour of a group of tasks from the rest of the system. $ mount | egrep "/cgroup |/memory" tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) $ lssubsys

CockroachDB performance benchmarking

CockroachDB key concepts * Range - CockroachDB stores all user data (tables, indexes, etc.) and almost all system data in a giant sorted map of key-value pairs. This keyspace is divided into "ranges", contiguous chunks of the keyspace, so that every key can always be found in a single range. From

Using cgroups to limit CPU utilization

Intro to cgroups Cgroups(control groups) make it possible to allocate system resources such as CPU time, memory, disk I/O and network bandwidth, or combinations of them, among a group of tasks(processes) running on a system. The following commands output the available subsystems(resource controllers) for the cgroups.

Deploy CockroachDB in kubernetes cluster

Deploy Kubernetes cluster $ kubectl get nodes NAME STATUS ROLES AGE VERSION node0 Ready 115d v1.19.2 node1 Ready 115d v1.19.2 node2 Ready 115d v1.19.2 node3 Ready master 115d v1.19.2 Start CockroachDB cluster Start the CockroachDB nodes with a configuration file that has been

Setup CockroachDB in docker container

Create docker volume Create docker volume on each host: [root@host1 ~]# docker volume create --driver local --opt type=ext4 --opt device=/dev/nvme2n1 vol1 [root@host1 ~]# mkfs.ext4 /dev/nvme2n1 [root@host1 ~]# docker inspect vol1 [ { "CreatedAt": "2022-09-13T01:10:50Z", "Driver": "local", "Labels": {}, "Mountpoint": "/var/lib/docker/volumes/vol1/_data", "Name"

Setup CockroachDB cluster with HAProxy load balancing

Each CockroachDB node is an equally suitable SQL gateway to your cluster, but to ensure client performance and reliability, it's important to use load balancing. * Performance: Load balancers spread client traffic across nodes. This prevents any one node from being overwhelmed by requests and improves overall cluster performance (queries per

How to delete partition in Linux

Identify the disk which contains the partitions $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 9.5G 0 rom nvme0n1 259:6 0 1.5T 0 disk ├─nvme0n1p1 259:7 0 200M 0 part /boot/efi ├─nvme0n1p2 259:8 0 1G 0 part /boot └─nvme0n1p3

Useful Elastic cluster APIs

Useful Elastic cluster APIs

Benchmarking Elasticsearch cluster with Rally

Install Rally on each node Prerequisites $ yum update $ yum install openssl-devel bzip2-devel libffi-devel $ yum groupinstall "Development Tools" Install Python 3.8+ $ wget https://www.python.org/ftp/python/3.8.15/Python-3.8.15.tar.xz $ tar xf Python-3.8.15.tar.xz $ vim Python-3.8.15/Modules/Setup SSL=

Getting started with Elastic Rally benchmark

Rally is the macrobenchmarking framework for Elasticsearch. This post follows the instructions here to install Rally and run the very first benchmark(aka race). Install python Python 3.8+ including pip3 is required for Rally. $ yum update $ yum install openssl-devel bzip2-devel libffi-devel $ yum groupinstall "Development Tools" $ curl -O https://www.

Overwrite the Elastic Rally track parameters

Each track allows to overwrite the corresponding parameters using --track-params. For example, the track geonames allows to overwrite the following parameters. * bulk_size (default: 5000) * bulk_indexing_clients (default: 8): Number of clients that issue bulk indexing requests. * ingest_percentage (default: 100): A number between 0 and 100 that defines

Key concepts in Elastic Rally

Pipelines A pipeline is a series of steps that are performed to get benchmark results. This is not intended to customize the actual benchmark but rather what happens before and after a benchmark. An example will clarify the concept: If you want to benchmark a binary distribution of Elasticsearch, Rally

Getting started with Elasticsearch and Kibana

Install elasticsearch Add user for elasticsearch [root@vm1 home]# groupadd es [root@vm1 home]# useradd es -g es [root@vm1 home]# passwd es [root@vm1 home]# cd es Download elasticsearch [root@vm1 es]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.4.3-linux-x86_64.tar.gz [root@vm1 es]# tar

How to setup the Grafana and Prometheus dashboard

Install node-exporter on each target nodes $ docker run --restart=always --name nd-export -d -p 9100:9100 -v /proc:/host/proc -v /sys:/host/sys -v /:/rootfs:ro,rslave --net=host quay.io/prometheus/node-exporter $ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d8c7fc8a4e40 quay.io/prometheus/node-exporter

Install golang from source

$ wget https://dl.google.com/go/go1.15.5.linux-amd64.tar.gz $ tar -C /usr/local -xvzf go1.15.5.linux-amd64.tar.gz $ cat /usr/local/go/VERSION go1.15.5 $ vim .bash_profile PATH=$PATH:$HOME/bin:/usr/local/go/bin export PATH $ source .bash_profile $ go version go

go-ycsb load db terminated with message 'Got signal [hangup] to exit'

During database load phase with go-ycsb, the load process is terminated and the message 'Got signal [hangup] to exit' is reported. $ nohup ./bin/go-ycsb load cockroach -P workloads/workloadd --threads 96 -p pg.host= -p pg.port=26257 -p pg.user=root -p pg.db=test -p pg.sslmode=disable

Limit usable memory size in Linux grub

In one case, I need to limit the usable memory size to 128GB from a system which has 1TB memory. $ cat /proc/meminfo | grep MemTotal MemTotal: 1056493068 kB Edit /etc/default/grub file Add "mem=128G" to the end of line "GRUB_CMDLINE_LINUX". $ cat /etc/default/grub GRUB_TIMEOUT=

YCSB performance benchmark on CockroachDB

Intro to go-ycsb go-ycsb is a Go port of YCSB. It fully supports all YCSB generators and the Core workload so we can do the basic CRUD benchmarks with Go. Install golang package $ wget https://dl.google.com/go/go1.19.2.linux-amd64.tar.gz $ tar -C /usr/local -xvzf

Understanding Azure disk bursting

Currently, there are two managed disk types that can burst, premium SSDs, and standard SSDs. Other disk types cannot currently burst. There are two models of bursting for disks: An on-demand bursting model, where the disk bursts whenever its needs exceed its current capacity. This model incurs additional charges anytime

Using netstat to check network package retransmission

$ while [ 1 ]; do netstat -s | grep 'segments retransmited' ; sleep 1; done 2500906 segments retransmited 2500912 segments retransmited 2501072 segments retransmited 2501102 segments retransmited 2501191 segments retransmited

Compile Linux kernel on CentOS

Build preparations To perform a successful kernel build, the following packages need to be installed: * yum groupinstall "Development Tools" * yum install ncurses-devel * yum install qt3-devel (This is only necessary if you wish to use "make xconfig" instead of "make gconfig" or "make menuconfig".) In the following example, we use "make

Deploy systemtap on multiple systems

Normally, SystemTap scripts can only be run on systems where SystemTap is deployed together with the following required kernel packages. * kernel-devel-$(uname -r) * kernel-debuginfo-$(uname -r) * kernel-debuginfo-common-$(uname -m)-$(uname -r) In the case that it's neither feasible nor desired to install the denpendent packages on all the target

Using systemtap to analyze latency of the kernel module function

In this post, we continue to explore how to use SystemTap to analyze the latency of the kernel module function. In the following example, we want to analyze the latency of the function "nfsd_vfs_write" from the kernel module "nfsd". Deploy the SystemTap packages Refer to this post to

Getting started with systemtap for Linux system profiling

Intro to SystemTap SystemTap is a tracing and probing tool that allows users to study and monitor the activities of the operating system (particularly, the kernel) in fine detail. It provides information similar to the output of tools like netstat, ps, top, and iostat; however, SystemTap is designed to provide

Using ftrace to analyze latency of the kernel module function

In this example, we study the latency of the function "nfsd_vfs_write" from kernel module "nfsd". ftrace configuration The following ftrace options are used in this example. There are 8 nfsd processes to be traced. * current_tracer: function_graph * pids: 3591 3592 3593 3594 3595 3596 3597 3598 * filters:

Dynamically tracing with user-defined tracepoint in perf

In this post, we are going to explore how to use perf for dynamic tracing with user-defined tracepoint. When we say dynamic tracing, the kernel event(function) to be traced is not predefined in perf. Instead, we add the tracepoint event manually. We will study it based on a real

tuned - dynamic adaptive system tuning daemon

Tuned Tuned is a daemon that uses udev to monitor connected devices and statically and dynamically tunes system settings according to a selected profile. Tuned is distributed with a number of predefined profiles for common use cases like high throughput, low latency, or powersave. It is possible to modify the

Install and uninstall debuginfo package on CentOS

Install debuginfo package On CentOS, we can install debuginfo package as below. * Modify /etc/yum.repos.d/CentOS-Debuginfo.repo by setting "enabled=1" * Run "yum install kernel-debuginfo" [root@host1 ~]# uname -r 3.10.0-1160.el7.x86_64 [root@host1 ~]# vim /etc/yum.repos.d/CentOS-Debuginfo.repo [base-debuginfo] name=CentOS-7 -

Using sar to understand system activity

About sadc and sar sadc is known as system activity data collector. It samples system data a specified number of times (count) at a specified interval measured in seconds (interval). It writes in binary format to the specified outfile or to standard output. The sadc command is intended to be

Using nconnect to improve NFS performance

The nfs mount option "nconnect=n" exists in all Linux distributions with kernel 5.3 or higher. nconnect enables multiple TCP connections for a single NFS mount. From the nfs manual page: nconnect=n When using a connection oriented protocol such as TCP, it may sometimes be advantageous to set

How to setup NFS (Network File System) on RHEL/CentOS

We need to install NFS packages on the NFS Server as well as on NFS Client machine. We can install it via “yum” for RHEL/CentOS. On the NFS server side, run the following commands to setup NFS server. $ systemctl status nfs $ mkfs.ext4 /dev/nvme9n1 $ mkdir /mnt/nfsshare $ vim

SysRq - Linux Magic System Request Key Hacks

What is the magic SysRq key? It is a ‘magical’ key combo you can hit which the kernel will respond to regardless of whatever else it is doing, unless it is completely locked up. How do I enable the magic SysRq key? You need to say “yes” to ‘Magic SysRq

journalctl - Query the systemd journal

journalctl may be used to query the contents of the systemd journal as written by systemd-journald.service. If called without parameters, it will show the full contents of the journal, starting with the oldest entry collected. The following are some of the common used options: * --no-full, --full, -l Ellipsize fields

Flame graph - A visualization for profiling and debugging

Flame graphs can be generated from the output of many different software profilers, including profiles for different resources and event types. The article written by Brendan Gregg describes how the flame graph works. We can start to work with the flame graph by following commands. $ yum install git $ git clone

A six-step framework to tackle any system design question

Introduction In the book of System Design Interview by Lewis C. Lin, a six-step framework called PEDALS is introduced to help answer any system design question. PEDALS stand for: * Process requirements * Estimate * Design the service * Articulate the data model * List the architectural components * Scale The systematic method will power you

The complete guide to System Design

The complete guide to System Design in 2022 covers: * What is System Design? * System Design fundamentals * Horizontal and vertical scaling * Microservices * Proxy servers * CAP theorem * Redundancy and replication * Storage * Block storage * File storage * Object storage * Redundant Disk Arrays (RAID) * Message queues * Kafka * File systems * Google File System (GFS) * Hadoop Distributed

Understanding RPO and RTO

The disasters could occur unexpectedly and it could take down the whole data center in an application region. In the real world, a well designed disaster recovery solution would help restore the applications and related data at the time of disaster. There are two critical metrics to measure how timely

Understanding the Linux inodes

Linux makes a clear distinction between the contents of a file and the information about a file. All information needed by the filesystem to handle a file is included in a data structure called an inode. Each file has its own inode, which the filesystem uses to identify the file.

Understanding MapReduce with an example

What is MapReduce MapReduce is a programming framework that allows us to perform distributed and parallel processing on large data sets in a distributed environment. * MapReduce consists of two distinct tasks — Map and Reduce. * As the name MapReduce suggests, reducer phase takes place after the mapper phase has been completed.

AngularJS table creation for dynamic number of columns

{{column}}

Hide and display html elements

Access a sharedv4 volume outside of a Portwrox cluster

To access a sharedv4 volume outside of the Portworx cluster, add the allow_ips label to the volume you wish to export, specifying a semi-colon separated list of IP addresses of non-Portworx nodes you wish to mount your sharedv4 volume to: $ /opt/pwx/bin/pxctl -j volume create datavol --sharedv4

Using iperf to measure the network bandwidth

iPerf is a tool for measurements of the maximum achievable bandwidth on IP network. The following iperf option can be used on the client side to saturate the network bandwidth if single client thread is not sufficient. * -P, --parallel # number of parallel client threads to run Benchmark a 100GbE network

How to deploy MySQL and phpMyAdmin with Docker

Create the MySQL docker container $ docker pull mysql/mysql-server:5.7 $ docker run -d --name=mysqldb -e MYSQL_ROOT_HOST=% -e MYSQL_ROOT_PASSWORD=password -e MYSQL_DATABASE=perfdb -v /var/lib/osd/mounts/mysql_data:/var/lib/mysql -p 3306:3306 mysql/mysql-server:5.7 $ docker logs mysqldb $ docker

What is Performance Engineering

Useful tools in Linux operating system

Performance tools * vmstat * iostat * ifstat * netstat * nfsstat * mpstat * nstat * dstat * sar * iftop * pidstat * xosview Benchmark tools * fio * iozone * iperf * netperf * vdbench * sysbench * pgbench * YCSB * SPEC SFS 2014/SPECstorage Solution 2020 * VMmark Debugging tools * htop * lslk * lsof * top Process tracing * ltrace * strace * pstack/gstack * ftrace * systemtap * ps * pmap * blktrace * ebpf Binary

OpenZFS - A pooled storage file system

OpenZFS The OpenZFS project is an open source derivative of the Oracle ZFS project. OpenZFS is an outstanding storage platform that encompasses the functionality of traditional filesystems, volume managers, and more, with consistent reliability, functionality and performance across all distributions. Source1 Source2 ZFS and ZFS Pooled Storage The ZFS file

Fio initial write, overwrite and append write

In this post, we try to study how the overwrite and file_append options work in fio benchmark. overwrite and file_append options * overwrite=bool If true, writes to a file will always overwrite existing data. If the file doesn’t already exist, it will be created before the write

Fio fsync, end_fsync, fdatasync and sync

Intro to fsync, end_fsync, fdatasync and sync * fsync=int If writing to a file, issue an fsync(2) (or its equivalent) of the dirty data for every number of blocks given. For example, if you give 32 as a parameter, fio will sync the file after every 32 writes

Scale a system to support millions of users

Start from single server Building a large scale system is not one time effort, it should be an iterative process as the user workload increases. The journey could just start with single server and single user request. 1. User access a website through domain name, such as example.com 2.

Set in golang

Go does not have set by default but there is way to imitate it in Go. The following is an example code piece to create "set" in Go by setting a map with empty values. $ cat test.go package main import ( "fmt" ) // Main function func main() { set := make(map[string]

Must-read books and articles for Software and Performance Engineering

Books * The Art of Computer Programming * Understanding the Linux Kernel, 3rd Edition * Linux Device Drivers, 3rd Edition * Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems, 1st Edition * Building Microservices: Designing Fine-Grained Systems 1st Edition * Site Reliability Engineering: How Google Runs Production Systems 1st E

Vdbench file system operations

Specify file system operations in FWD and RD In the File system Workload Definition(FWD), we can specify a single file sysetm operation that must be executed for the workload, with the parameter 'operation='. The valid operation can be mkdir,rmdir,create,delete,open,close,read,write,getattr and

Proc filesystem - diskstats

The /proc/diskstats file displays the I/O statistics of block devices. Here we have a system which has one disk sda used by Linux operating system and two disks sdb sdc for other purpose. $ cat /etc/centos-release CentOS Linux release 7.9.2009 (Core) $ uname -r 3.10.0-1160.

dstat - A replacement for vmstat, iostat and ifstat

Introduction dstat is a versatile tool for generating system resource statistics. It can be a replacement for vmstat, iostat and ifstat. It overcomes some of the limitations and adds some extra features. Dstat allows you to view all of your system resources instantly, you can eg. compare disk usage in

Docker-based fio benchmarking - Part Two

In this article, we will build a docker image which is based on python versioned alpine Linux. Alpine Linux is much smaller than most distribution base images (~5MB), and thus leads to much slimmer images in general. In the python versioned alpine Linux image, we can add additional packages to

Docker-based fio benchmarking - Part One

Why Docker-based fio benchmarking fio is a flexible I/O tester which generates I/O and measure I/O performance on the target storage system. In the case we want to run the fio workload on the cloud deployments, we can containerize fio. Also we can encapsulate necessary packages in

Concepts you should know

Data structure Bloom filter A Bloom filter is a space-efficient probabilistic data structure that is used to test whether an element is a member of a set. For example, checking availability of username is set membership problem, where the set is the list of all registered username. The price we

Storage concepts you should know

Object Storage Object storage, also known as object-based storage, is a strategy that manages and manipulates data storage as distinct units, called objects. These objects are kept in a single storehouse and are not ingrained in files inside other folders. Instead, object storage combines the pieces of data that make

Network concepts you should know

IP address Networking is about one computer sending a message to another computer. This message is called packet in the IP world. It is just like a postcard in the postal service. The postal service can take a postcard addressed to someone and deliver it. Similarly, it needs to provide

Database concepts you should know

SQL vs. NoSQL Source Benefits of NoSQL databases NoSQL databases offer many benefits over relational databases. NoSQL databases have flexible data models, scale horizontally, have incredibly fast queries, and are easy for developers to work with. * Flexible data models NoSQL databases typically have very flexible schemas. A flexible schema allows

Distributed system concepts you should know

This post contains basic explanations for concepts you should know related to distributed system. API and REST API Application Programming Interface, abbreviated as API, enables connection between computers or computer programs. It is a Software Interface that offers services to other software to enhance the required functionalities. REST API is

Estimate the scale of the system

It is always a good idea to estimate the scale of the system as it help reflect if the designed system could fulfill the functional requirements. The requirements might include: * Number of users * Number of active users(NAU) * Requrests per second(RPS) * Logins per seconds * Transactions per second(TPS) for

What is high availability

What is high availability? High availability refers to a system or component that is operational without interruption for long periods of time. High availability is measured as a percentage, with a 100% percent system indicating a service that experiences zero downtime. This would be a system that never fails. It’

Latency numbers

Latency Comparison Numbers (~2012) Conclusion * Disk is much slower than the memory * Avoid the disk seek if possible * Compressing data is worth to consider before sending over network * It takes time to send data between data centers if they are in different regions Reference * https://computers-are-fast.github.io/ * http://research.

Data volume unit

Data volumes The volume of data in a single file or file system can be described by a unit called a byte. However, data volumes can become very large when dealing with Earth satellite data. Below is a table to explain data volume units (Credit: Roy Williams, Center for Advanced

Install git from source

$ wget https://mirrors.edge.kernel.org/pub/software/scm/git/git-2.38.1.tar.xz $ tar xvf git-2.38.1.tar.xz $ cd git-2.38.1/ $ make configure $ ./configure --prefix=/usr/local $ make install $ git --version git version 2.38.1 Reference * https://mirrors.edge.kernel.org/pub/software/scm/

Understanding thin provisioning volume and snapshot

Thin provisioning volume Logical volume can be thinly provisioned. It allows storage administrator to overcommit the physical storage. In other words, it's possible to create a logical volume which is larger than the available extents. Create thin provisioning volume In the following example, we create a 500GiB thin pool and

How to configure SAR data collection on RHEL8

On RHEL8, it uses systemd instead of cron jobs to manage SAR data collection service. Run the following command to check if the SAR data collection is started. [root@h04-11 ~]# cat /etc/redhat-release Red Hat Enterprise Linux release 8.5 (Ootpa) [root@h04-11 ~]# systemctl status sysstat-collect.timer ● sysstat-collect.timer -

Solve LeetCode problems in Visual Studio Code

Install the leetcode extension in VS Code Login to LeetCode endpoint with cookie 1. Login to leetcode from Google Chrome 2. In Chrome, Inspect -> Network -> Fetch/XHR 3. Click on any button in leetcode page, and in Inspector to the right under the "Name" tab find and select

libaio init failed due to resource temporarily unavailable

Issue Description $ sudo fio --blocksize=64k --directory=/mnt/bench1 --filename=testfile --ioengine=libaio --readwrite=randread --size=10G --name=test --numjobs=512 --group_reporting --direct=1 --iodepth=128 --randrepeat=1 --disable_lat=0 --gtod_reduce=0 test: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T)

How to uninstall ceph storage cluster

Check the pools, images and OSDs [ceph: root@host1 /]$ ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 83.83411 root default -3 27.94470 host host1 0 ssd 3.49309 osd.0 up 1.00000 1.00000 1 ssd 3.49309 osd.1 up 1.00000

fio benchmark on multiple files

fio directory and filename options To run fio benchmark on multiple files or deives, we should understand the following fio options. * directory=str Prefix filenames with this directory. Used to place files in a different location than ./. You can specify a number of directories by separating the names with a

fio benchmark on multiple devices

In this post, we study how to run fio benchmark on multiple devices. We also try to understand how the iodepth reflects on each device. We start with single device and the following global parameters are used. * blocksize=16k * filesize=50G (write/read 50G data on each device) * iodepth=64

Backup and restore MySQL database

Backup a database mysqldump is a command-line utility which can be used to generate backups of MySQL database. $ mysqldump -u root --password= mydb > mydb_dump_`date +"%Y%m%d_%H%M%S"`.sql $ ls -ltr | grep mydb -rw-r--r--. 1 root root 4834575 Sep 28 21:11 mydb_dump_20210928_144610.

Create Image Gallery in Jekyll without Plugin

An image gallery can be easily built by using LightBox and Image Gallery scripts in Jekyll. 1. LightBox Lightbox is a solution that loads your image links, your Youtube links and your Vimeo links automatically in a minimalistic and responsive pseudo window/overlay. No adjustment to your links is required,

Using sysbench for OLTP workload performance benchmark

Intro to Sysbench sysbench is a scriptable multi-threaded benchmark tool based on LuaJIT. It is most frequently used for database benchmarks, but can also be used to create arbitrarily complex workloads that do not involve a database server. sysbench comes with the following bundled benchmarks: * oltp_*.lua: a collection of

An easy guide to install kubernetes cluster with kubeadm

Kubernetes can be installed with the following deployments tools. * Bootstrapping clusters with kubeadm * Installing Kubernetes with kops * Installing Kubernetes with Kubespray In this article, we learn to install Kubernetes cluster with kubeadm. Installing kubeadm Prepare the cluster nodes We have three Centos nodes to install Kubernetes cluster. $ cat /etc/centos-release

Vdbench performance test on filesystem

File system testing - Method 1 Vdbench filesystem testing terminologies: * Anchor - A directory or a filesystme mount point. A file system structure will be created by specifying the structure information including directory depth, width, number of files and file size. Multiple anchor can be defined and used by filesystem

Vdbench performance test on raw device

Terminology * Master and Slave: Vdbench runs as two or more Java Virtual Machines (JVMs). The JVM that you start is the master. The master takes care of the parsing of all the parameters, it determines which workloads should run, and then will also do all the reporting. The actual workload

Deploy ceph cluster on Ubuntu 18.04 and CentOS 7.8

In this article, we learn to deploy ceph cluster on ubuntu 18.04. Three nodes are used for this study. We target to deploy the most recent ceph release which is called Pacific. With this release, we can use cephadm to create a ceph cluster by bootstrapping on a single

Capture and analyze network packets with tcpdump

Capture packets with tcpdump In this example, we only capture 1000 packets(-c1000) and use IP addresses and ports(-nn) for easier analysis. The raw packets are written to file "tcpdump.1000" for further analsis. $ tcpdump -i eth5 -c1000 -nn -w tcpdump.1000 View the packets data We can use

Useful tools to analyze network latency

ping Ping is one of the most basic commands in network management, verifying network connectivity through the roundtrip times taken by the ICMP protocol packets sent to a target host. ping - send ICMP ECHO_REQUEST to network hosts * -c count Stop after sending count ECHO_REQUEST packets. With deadline

Iouring - A modern asynchronous I/O interface for Linux

The Linux kernel has had asynchronous I/O since version 2.5, but it was seen as difficult to use and inefficient. io_uring (previously known as aioring) is a Linux kernel system call interface for storage device asynchronous I/O operations addressing performance issues with similar interfaces provided by

btrfs - A modern copy on write(CoW) filesystem for Linux

Introduction btrfs is a modern copy on write (CoW) filesystem for Linux aimed at implementing advanced features while also focusing on fault tolerance, repair and easy administration. Its main features and benefits are: * Snapshots which do not make the full copy of files * RAID - support for software-based RAID 0,

Setting up LVM volumes on a mdraid array

Setting up mdraid array $ mdadm --create /dev/md0 --name=mdvol --level=raid0 --raid-devices=2 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md0 started. $ cat /proc/mdstat Personalities : [raid0] [linear] md0 : active raid0 nvme4n1[3] nvme3n1[2] nvme2n1[1] nvme1n1[

Linux Software RAID

What is RAID? RAID stands for either Redundant Array of Independent Disks, or Redundant Array of Inexpensive Disks. The intention of RAID is to spread your data across several disks, such that a single disk failure will not lose that data. The current RAID drivers in Linux support the following

RAID - Redundant Array of Inexpensive Disks

RAID stands for Redundant Array of Inexpensive (Independent) Disks. On most situations you will be using one of the following four levels of RAIDs. * RAID 0 * RAID 1 * RAID 5(6) * RAID 10 (also known as RAID 1+0) RAID 0 Following are the key points to remember for RAID

Upgrade kernel version in centos 7

Before upgrade $ cat /etc/centos-release CentOS Linux release 7.4.1708 (Core) $ uname -r 3.10.0-693.el7.x86_64 $ cat /boot/grub2/grubenv saved_entry=CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core) Install the newer kernel package $ rpm -ivh kernel-ml-5.7.12-1.el7.elrepo.x86_64.

Migrate docker data to different directory

This post guides us how to migrate the docker data from the existing directory to a target directory in the case that the old directory runs out of space. Stop the docker daemon $ cat /etc/centos-release CentOS Linux release 7.9.2009 (Core) $ systemctl stop docker.service $ ps aux | grep

Using gstack for docker container process

In a Docker container environment, we won't get a valid stack trace directly on the container host as below. $ ps -ef |grep smbd root 171118 167977 0 Apr22 ? 00:00:02 /usr/sbin/smbd --foreground --no-process-group root 171166 171118 0 Apr22 ? 00:00:00 /usr/sbin/smbd --foreground --no-process-group root

Network bonding

Network bonding enables the combination of two or more network interfaces into a single-bonded (logical) interface, which increases the bandwidth and provides redundancy. If a specific network interface card (NIC) experiences a problem, communications are not affected significantly as long as the other slave NICs remain active. Bonding modes supported

Enable ssh passwordless login for large amount of servers

Install sshpass package To install sshpass package on the source server from which to enable ssh passwordless login to remote servers: -bash-4.2# rpm -ivh sshpass-1.06-1.el7.x86_64.rpm Enable ssh passwordless login to remote servers To enable ssh passwordless login to a large amount of remote servers,

Tuning kernel parameters in Docker

Configure namespaced kernel parameters(sysctl) at runtime The --sysctl sets namespaced kernel parameters (sysctls) in the container. For example, to turn on IP forwarding in the containers network namespace, run this command: $ docker run --sysctl net.ipv4.ip_forward=1 someimage Note Not all sysctls are namespaced. Docker does not

Sorting algorithm

In this post, it includes the following sorting algorithms. The code is self explained. * selection sort * insert sort * bubble sort * quick sort * merge sort import java.util.Arrays; public class Sort { public static void selection_sort(int[] arr) { for (int i = 0; i < arr.length - 1; i++) { int min_

Thread synchronization

Mutexes A mutex is basically a lock that we set (lock) before accessing a shared resource and release (unlock) when we’re done. While it is set, any other thread that tries to set it will block until we release it. If more than one thread is blocked when we

Interprocess communication(IPC)

Pipes Pipes are the oldest form of UNIX System IPC and are provided by all UNIX systems. Pipes have two limitations. 1. Historically, they have been half duplex (i.e., data flows in only one direction). Some systems now provide full-duplex pipes, but for maximum portability, we should never assume

Using bcc-tools for dynamic kernel tracing

In Red Hat Enterprise Linux 8.1, Red Hat ships a set of fully supported on x86_64 dynamic kernel tracing tools, called bcc-tools, that make use of a kernel technology called extended Berkeley Packet Filter (eBPF). With these tools, you can quickly gain insight into certain aspects of system

blktrace - A block layer IO tracing utility

What can blktrace do? Let's dive into the block device layer and see how the I/O are handled in disk queues. The following stack shows the I/O paths including the block device layer. The application can issue I/O requests directly to block device or through file systems.

Understanding await in iostat

What's meaning of await in iostat? The following is the description provided for await field in iostat man page. $ man iostat await The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the

Buffered and Direct IO

Buffered and Direct I/O The VxFS responds with read-ahead for sequential read I/O. This results in buffered I/O. The data is prefetched and retained in buffers, in anticipation of application asking for it. The data buffers are commonly referred to as the VxFS buffer cache. This is

Network ring buffer

Introduction Receive ring buffers are shared between the device driver and NIC. The card assigns a transmit (TX) and receive (RX) ring buffer. As the name implies, the ring buffer is a circular buffer where an overflow simply overwrites existing data. It should be noted that there are two ways

Semaphore

What is semaphore A semaphore is a very relaxed type of lockable object. A given semaphore has a predefined maximum count, and a current count. You take ownership of a semaphore with a wait operation, also referred to as decrementing the semaphore, or even just abstractly called P. You release

Max open files limit

System wide open files limit To check system wide files limit: $ cat /proc/sys/fs/file-max 4875932 $ sysctl -a | grep file-max fs.file-max = 4875932 To change system wide files limit: $ echo "fs.file-max = 4875932" >> /etc/sysctl.conf $ sysctl -p /etc/sysctl.conf User level open files limit To check hard/

Iozone - A filesystem benchmark tool

Introduction IOzone is a filesystem benchmark tool. The benchmark generates and measures a variety of file operations. Iozone is useful for performing a broad filesystem analysis of a vendor’s computer platform. The benchmark tests file I/O performance for the following operations: * Read, write, re-read, re-write, read backwards, read

Kubernetes and Gluster performance

Kubernetes and Gluster Intro Kubernetes, also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications. Gluster is a scalable, distributed file system that aggregates disk storage resources from multiple servers into a single global namespace. Gluster performance study In this article, we will

GlusterFS - A distributed file syste

What is Gluster? Gluster is a scalable, distributed file system that aggregates disk storage resources from multiple servers into a single global namespace. Advantages * Scales to several petabytes * Handles thousands of clients * POSIX compatible * Uses commodity hardware * Can use any ondisk filesystem that supports extended attributes * Accessible using industry standard

CBT - Changed Block Tracking

Introduction Changed Block Tracking is an incremental backup technology for virtual machines. It helps create faster and smaller backups. It has the following advantages. * Reduce backup time * Save disk space by only storing changed data to the previous backup Block changes are tracked in the virtualization layer, outside the virtual

Perf - The official Linux profiler

Introduction perf, aka perf_events, is the official Linux profiler and included in the Linux kernel source under tools/perf. It can instrument CPU performance counters, tracepoints, kprobes, and uprobes. It is capable of system profiling. Performance Monitoring Counter(PMC) Show system wide PMC statistics The following example shows PMC

Using fio to generate millions of files

fio is the acronym for Flexible IO Tester and is a tool for I/O performance measurement. From time to time, I use fio to run performance benchmark test, either to investigate I/O performance for certain storage or reproduce performance issues. In this article, we are not target to

Analyze the library and system calls

When the application source code is not available, we can trace the function calls during its runtime in order to understand how the application code works. Library calls take place in user space. System calls is the fundamental interface between application and the kernel. System calls are not invoked directly

Memory-Mapped I/O

Introduction Memory-mapped I/O lets us map a file on disk into a buffer in memory so that, when we fetch bytes from the buffer, the corresponding bytes of the file are read. Similarly, when we store data in the buffer, the corresponding bytes are automatically written to the file.

Memory fragmentation

Memory fragmentation Memory page availability can be checked from /proc/buddyinfo as below. $ cat /proc/buddyinfo Node 0, zone DMA 1 0 1 0 1 1 1 0 1 1 3 Node 0, zone DMA32 3342 2441 2138 5025 1871 236 3 0 0 0 0 Node 0, zone Normal

Memory overcommit

The Linux kernel supports the following overcommit handling modes 0 - Heuristic overcommit handling. Obvious overcommits of address space are refused. Used for a typical system. It ensures a seriously wild allocation fails while allowing overcommit to reduce swap usage. root is allowed to allocate slightly more memory in this

kmalloc and vmalloc

kmalloc The function allocates contiguous region in physical memory. It's fast and doesn't clear the allocated memory content. kmalloc function is defined in include/linux/slab.h /** * kmalloc - allocate memory * @size: how many bytes of memory are required. * @flags: the type of memory to allocate. * * kmalloc is the normal

Useful HBA commands

Check HBA card type $ lspci | grep -i fibre 18:00.0 Fibre Channel: QLogic Corp. ISP2722-based 16/32Gb Fibre Channel to PCIe Adapter (rev 01) 18:00.1 Fibre Channel: QLogic Corp. ISP2722-based 16/32Gb Fibre Channel to PCIe Adapter (rev 01) Physical slot, driver, module information $ lspci -v -s

Monitor RMAN backup and restore progress

The following script will allow you to monitor progress of an RMAN backup or Restore from 4 different perspectives (channel, session wait events, datafiles, backuppieces). The script is run from SQLPlus and takes a date input value in the format ‘dd-mon-rr hh24:mi:ss’. The date supplied does not have

RMAN backup set, pieces, and datafiles

RMAN backs up the datafiles, control file, archived log files, and server parameter files in a RMAN specific format called a backup piece. In a nutshell, a backup set is a bundle of dbf, ctl and redo file that can restore a database. A set of one or more such

Level 0 and Level 1 Incremental Backups

Incremental backups can be either level 0 or level 1. A level 0 incremental backup, which is the base for subsequent incremental backups, copies all blocks containing data, backing the datafile up into a backup set just as a full backup would. The only difference between a level 0 incremental

What is patrol read

What is patrolread The Patrol Read feature is designed as a preventative measure to ensure hard drive health and data integrity. Patrol Read scans for and resolves potential problems on configured hard drives. Patrol Read Behavior The following is an overview of Patrol Read behavior: * Patrol Read runs on all

Shell, sed and awk scripting

Shell Special variables in Shell: * $0 - The filename of the current script. * $# - The number of arguments supplied to a script. * $* - All the arguments are double quoted. If a script receives two arguments, $* is equivalent to $1 $2. * $@ - All the arguments are individually double quoted. If a

Linux System Administration

Package management Install the package: $ rpm -ivh blktrace Upgrade the package: $ rpm -Uvh blktrace Remove the installed package: $ rpm -ev blktrace $ rpm -ev --nodeps blktrace Display the installed package info: $ rpm -qi blktrace Name : blktrace Relocations: (not relocatable) Version : 1.0.1 Vendor: Red Hat, Inc. Release : 6.el6 Build

Sublime Jekyll Package

sublime-jekyll is a Sublime Text package for Jekyll static sites. This package is aimed at streamlining the process of managing and creating new Jekyll posts. It does this by providing quick and easy access to new post and draft commands, snippets and completions for Liquid template tags and filters, as

Start jekyll with systemd service

Create systemd file $ cat /etc/systemd/system/jekyll.service # Description: to make `jekyll serve` a system service and start on boot # # Usage: place this file at `/etc/systemd/system/jekyll.service` # then run # sudo systemctl start jekyll.service # sudo systemctl enable jekyll.service [Unit] Description=Jekyll service After=syslog.target

Install Jekyll on MacOS

Install Command Line Tools First, you need to install the command-line tools to be able to compile native extensions, open a terminal and run: $ xcode-select --install Install Ruby Jekyll requires Ruby > 2.5.0. macOS Catalina 10.15 comes with ruby 2.6.3, so you’re fine. If you’

Transform Plain Text Into Static Websites And Blogs

Quick start instructions $ gem install bundler jekyll $ jekyll new my-awesome-site $ cd my-awesome-site $ bundle exec jekyll serve # => Now browse to http://localhost:4000 Check out the [Jekyll][jekyll] for more info on how to translate your plain text into static websites and blogs. Reference * https://jekyllrb.com/