Linux

Using cgroup to limit resource for ssh session

/etc/profile and /etc/profile.d Before we discuss how to limit process resource usage with cgroup for ssh session, let's take a moment to know about /etc/profile and /etc/profile.d in Linux. /etc/profile This is the system-wide profile file, which will be executed, when a user

Using dm-delay to emulate slower SSD on NVME device

Intro to dm-delay Device-Mapper's "delay" target delays reads and/or writes and maps them to different devices. Setup dm-delay To setup a dm-delay for extra 1ms latency: $ cat setup_dm_delay.sh disk=nvme2n1 size=$(blockdev --getsize /dev/$disk) echo "0 $size delay /dev/$disk 0 1" | dmsetup create delayed2

TCP 3-way handshake (SYN, SYN-ACK,ACK)

What is TCP Three-Way HandShake TCP 3-way handshake is a process which is used in a TCP/IP network to make a connection between the client and server. It is a three-step process that requires both the client and server to exchange synchronization and acknowledgment packets before the real data

Difference between soft links and hard links

Intro A hard link acts as a copy (mirrored) of the selected file. It accesses the data available in the original file. If the earlier selected file is deleted, the hard link to the file will still contain the data of that file. A soft link (aka Symbolic link) acts

Emulate a slow block device with dm-delay

It might be necessary to emulate a slow block device when you don't have a slow disk for debugging some high latency performance issue. The device mapper driver in Linux provides a solution called "delay" target. In this post, we will learn how to use the "delay" target to delay

fsstat - Display details of a file system

fsstat displays the details associated with a file system. The output of this command is file system specific. At a minimum, the range of meta-data values (inode numbers) and content units (blocks or clusters) are given. Also given are details from the Super Block, such as mount times and and

Upgrade Linux kernel on CentOS 7

Before upgrade $ cat /etc/centos-release CentOS Linux release 7.4.1708 (Core) $ uname -r 3.10.0-693.el7.x86_64 $ cat /boot/grub2/grubenv saved_entry=CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core) Install the newer kernel package $ rpm -ivh kernel-ml-5.7.12-1.el7.elrepo.x86_64.

How to install BCC on Ubuntu 22.04

To build the toolchain from source, one needs: LLVM 3.7.1 or newer, compiled with BPF support (default=on) Clang, built from the same tree as LLVM cmake (>=3.1), gcc (>=4.7), flex, bison LuaJIT, if you want Lua support root@ubuntu:~# cat /etc/*release DISTRIB_ID=Ubuntu

Install GCC from source

In this post, we will go through the steps to install GCC.

How to install GUI desktop on CentOS 7

Install GNOME desktop via the groups option of yum: $ yum update $ yum -y groups install "GNOME Desktop" Inform the startx command which desktop env to run: $ echo "exec gnome-session" >> ~/.xinitrc Manually start the GUI desktop: $ startx Automaticlly start the GUI desktop after reboot: $ systemctl set-default graphical.target $ reboot

Using signals with kill command in Linux

Usage of signals with kill $ kill -l 1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP 6) SIGABRT 7) SIGBUS 8) SIGFPE 9) SIGKILL 10) SIGUSR1 11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM 16) SIGSTKFLT 17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP 21) SIGTTIN 22)

Using cgroups to limit block device bandwidth

The block I/O controller specifies upper IO rate limits on devices. $ mount | egrep "/cgroup |/blkio" tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) $ lscgroup | grep blkio blkio:/ In this post,

Using cgroups to limit Memory usage

The memory controller isolates the memory behaviour of a group of tasks from the rest of the system. $ mount | egrep "/cgroup |/memory" tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) $ lssubsys

Using cgroups to limit CPU utilization

Intro to cgroups Cgroups(control groups) make it possible to allocate system resources such as CPU time, memory, disk I/O and network bandwidth, or combinations of them, among a group of tasks(processes) running on a system. The following commands output the available subsystems(resource controllers) for the cgroups.

How to delete partition in Linux

Identify the disk which contains the partitions $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 9.5G 0 rom nvme0n1 259:6 0 1.5T 0 disk ├─nvme0n1p1 259:7 0 200M 0 part /boot/efi ├─nvme0n1p2 259:8 0 1G 0 part /boot └─nvme0n1p3

Install golang from source

$ wget https://dl.google.com/go/go1.15.5.linux-amd64.tar.gz $ tar -C /usr/local -xvzf go1.15.5.linux-amd64.tar.gz $ cat /usr/local/go/VERSION go1.15.5 $ vim .bash_profile PATH=$PATH:$HOME/bin:/usr/local/go/bin export PATH $ source .bash_profile $ go version go

Limit usable memory size in Linux grub

In one case, I need to limit the usable memory size to 128GB from a system which has 1TB memory. $ cat /proc/meminfo | grep MemTotal MemTotal: 1056493068 kB Edit /etc/default/grub file Add "mem=128G" to the end of line "GRUB_CMDLINE_LINUX". $ cat /etc/default/grub GRUB_TIMEOUT=

Using netstat to check network package retransmission

$ while [ 1 ]; do netstat -s | grep 'segments retransmited' ; sleep 1; done 2500906 segments retransmited 2500912 segments retransmited 2501072 segments retransmited 2501102 segments retransmited 2501191 segments retransmited

Compile Linux kernel on CentOS

Build preparations To perform a successful kernel build, the following packages need to be installed: * yum groupinstall "Development Tools" * yum install ncurses-devel * yum install qt3-devel (This is only necessary if you wish to use "make xconfig" instead of "make gconfig" or "make menuconfig".) In the following example, we use "make

Deploy systemtap on multiple systems

Normally, SystemTap scripts can only be run on systems where SystemTap is deployed together with the following required kernel packages. * kernel-devel-$(uname -r) * kernel-debuginfo-$(uname -r) * kernel-debuginfo-common-$(uname -m)-$(uname -r) In the case that it's neither feasible nor desired to install the denpendent packages on all the target

tuned - dynamic adaptive system tuning daemon

Tuned Tuned is a daemon that uses udev to monitor connected devices and statically and dynamically tunes system settings according to a selected profile. Tuned is distributed with a number of predefined profiles for common use cases like high throughput, low latency, or powersave. It is possible to modify the

Install and uninstall debuginfo package on CentOS

Install debuginfo package On CentOS, we can install debuginfo package as below. * Modify /etc/yum.repos.d/CentOS-Debuginfo.repo by setting "enabled=1" * Run "yum install kernel-debuginfo" [root@host1 ~]# uname -r 3.10.0-1160.el7.x86_64 [root@host1 ~]# vim /etc/yum.repos.d/CentOS-Debuginfo.repo [base-debuginfo] name=CentOS-7 -

Using sar to understand system activity

About sadc and sar sadc is known as system activity data collector. It samples system data a specified number of times (count) at a specified interval measured in seconds (interval). It writes in binary format to the specified outfile or to standard output. The sadc command is intended to be

Using nconnect to improve NFS performance

The nfs mount option "nconnect=n" exists in all Linux distributions with kernel 5.3 or higher. nconnect enables multiple TCP connections for a single NFS mount. From the nfs manual page: nconnect=n When using a connection oriented protocol such as TCP, it may sometimes be advantageous to set

How to setup NFS (Network File System) on RHEL/CentOS

We need to install NFS packages on the NFS Server as well as on NFS Client machine. We can install it via “yum” for RHEL/CentOS. On the NFS server side, run the following commands to setup NFS server. $ systemctl status nfs $ mkfs.ext4 /dev/nvme9n1 $ mkdir /mnt/nfsshare $ vim

SysRq - Linux Magic System Request Key Hacks

What is the magic SysRq key? It is a ‘magical’ key combo you can hit which the kernel will respond to regardless of whatever else it is doing, unless it is completely locked up. How do I enable the magic SysRq key? You need to say “yes” to ‘Magic SysRq

journalctl - Query the systemd journal

journalctl may be used to query the contents of the systemd journal as written by systemd-journald.service. If called without parameters, it will show the full contents of the journal, starting with the oldest entry collected. The following are some of the common used options: * --no-full, --full, -l Ellipsize fields

Understanding the Linux inodes

Linux makes a clear distinction between the contents of a file and the information about a file. All information needed by the filesystem to handle a file is included in a data structure called an inode. Each file has its own inode, which the filesystem uses to identify the file.

Proc filesystem - diskstats

The /proc/diskstats file displays the I/O statistics of block devices. Here we have a system which has one disk sda used by Linux operating system and two disks sdb sdc for other purpose. $ cat /etc/centos-release CentOS Linux release 7.9.2009 (Core) $ uname -r 3.10.0-1160.

dstat - A replacement for vmstat, iostat and ifstat

Introduction dstat is a versatile tool for generating system resource statistics. It can be a replacement for vmstat, iostat and ifstat. It overcomes some of the limitations and adds some extra features. Dstat allows you to view all of your system resources instantly, you can eg. compare disk usage in

Network concepts you should know

IP address Networking is about one computer sending a message to another computer. This message is called packet in the IP world. It is just like a postcard in the postal service. The postal service can take a postcard addressed to someone and deliver it. Similarly, it needs to provide

Install git from source

$ wget https://mirrors.edge.kernel.org/pub/software/scm/git/git-2.38.1.tar.xz $ tar xvf git-2.38.1.tar.xz $ cd git-2.38.1/ $ make configure $ ./configure --prefix=/usr/local $ make install $ git --version git version 2.38.1 Reference * https://mirrors.edge.kernel.org/pub/software/scm/

Understanding thin provisioning volume and snapshot

Thin provisioning volume Logical volume can be thinly provisioned. It allows storage administrator to overcommit the physical storage. In other words, it's possible to create a logical volume which is larger than the available extents. Create thin provisioning volume In the following example, we create a 500GiB thin pool and

How to configure SAR data collection on RHEL8

On RHEL8, it uses systemd instead of cron jobs to manage SAR data collection service. Run the following command to check if the SAR data collection is started. [root@h04-11 ~]# cat /etc/redhat-release Red Hat Enterprise Linux release 8.5 (Ootpa) [root@h04-11 ~]# systemctl status sysstat-collect.timer ● sysstat-collect.timer -

Capture and analyze network packets with tcpdump

Capture packets with tcpdump In this example, we only capture 1000 packets(-c1000) and use IP addresses and ports(-nn) for easier analysis. The raw packets are written to file "tcpdump.1000" for further analsis. $ tcpdump -i eth5 -c1000 -nn -w tcpdump.1000 View the packets data We can use

Useful tools to analyze network latency

ping Ping is one of the most basic commands in network management, verifying network connectivity through the roundtrip times taken by the ICMP protocol packets sent to a target host. ping - send ICMP ECHO_REQUEST to network hosts * -c count Stop after sending count ECHO_REQUEST packets. With deadline

Iouring - A modern asynchronous I/O interface for Linux

The Linux kernel has had asynchronous I/O since version 2.5, but it was seen as difficult to use and inefficient. io_uring (previously known as aioring) is a Linux kernel system call interface for storage device asynchronous I/O operations addressing performance issues with similar interfaces provided by

Setting up LVM volumes on a mdraid array

Setting up mdraid array $ mdadm --create /dev/md0 --name=mdvol --level=raid0 --raid-devices=2 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md0 started. $ cat /proc/mdstat Personalities : [raid0] [linear] md0 : active raid0 nvme4n1[3] nvme3n1[2] nvme2n1[1] nvme1n1[

Linux Software RAID

What is RAID? RAID stands for either Redundant Array of Independent Disks, or Redundant Array of Inexpensive Disks. The intention of RAID is to spread your data across several disks, such that a single disk failure will not lose that data. The current RAID drivers in Linux support the following

Upgrade kernel version in centos 7

Before upgrade $ cat /etc/centos-release CentOS Linux release 7.4.1708 (Core) $ uname -r 3.10.0-693.el7.x86_64 $ cat /boot/grub2/grubenv saved_entry=CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core) Install the newer kernel package $ rpm -ivh kernel-ml-5.7.12-1.el7.elrepo.x86_64.

Network bonding

Network bonding enables the combination of two or more network interfaces into a single-bonded (logical) interface, which increases the bandwidth and provides redundancy. If a specific network interface card (NIC) experiences a problem, communications are not affected significantly as long as the other slave NICs remain active. Bonding modes supported

Enable ssh passwordless login for large amount of servers

Install sshpass package To install sshpass package on the source server from which to enable ssh passwordless login to remote servers: -bash-4.2# rpm -ivh sshpass-1.06-1.el7.x86_64.rpm Enable ssh passwordless login to remote servers To enable ssh passwordless login to a large amount of remote servers,

Thread synchronization

Mutexes A mutex is basically a lock that we set (lock) before accessing a shared resource and release (unlock) when we’re done. While it is set, any other thread that tries to set it will block until we release it. If more than one thread is blocked when we

Interprocess communication(IPC)

Pipes Pipes are the oldest form of UNIX System IPC and are provided by all UNIX systems. Pipes have two limitations. 1. Historically, they have been half duplex (i.e., data flows in only one direction). Some systems now provide full-duplex pipes, but for maximum portability, we should never assume

Network ring buffer

Introduction Receive ring buffers are shared between the device driver and NIC. The card assigns a transmit (TX) and receive (RX) ring buffer. As the name implies, the ring buffer is a circular buffer where an overflow simply overwrites existing data. It should be noted that there are two ways

Semaphore

What is semaphore A semaphore is a very relaxed type of lockable object. A given semaphore has a predefined maximum count, and a current count. You take ownership of a semaphore with a wait operation, also referred to as decrementing the semaphore, or even just abstractly called P. You release

Max open files limit

System wide open files limit To check system wide files limit: $ cat /proc/sys/fs/file-max 4875932 $ sysctl -a | grep file-max fs.file-max = 4875932 To change system wide files limit: $ echo "fs.file-max = 4875932" >> /etc/sysctl.conf $ sysctl -p /etc/sysctl.conf User level open files limit To check hard/

Memory-Mapped I/O

Introduction Memory-mapped I/O lets us map a file on disk into a buffer in memory so that, when we fetch bytes from the buffer, the corresponding bytes of the file are read. Similarly, when we store data in the buffer, the corresponding bytes are automatically written to the file.

Memory overcommit

The Linux kernel supports the following overcommit handling modes 0 - Heuristic overcommit handling. Obvious overcommits of address space are refused. Used for a typical system. It ensures a seriously wild allocation fails while allowing overcommit to reduce swap usage. root is allowed to allocate slightly more memory in this

kmalloc and vmalloc

kmalloc The function allocates contiguous region in physical memory. It's fast and doesn't clear the allocated memory content. kmalloc function is defined in include/linux/slab.h /** * kmalloc - allocate memory * @size: how many bytes of memory are required. * @flags: the type of memory to allocate. * * kmalloc is the normal

Useful HBA commands

Check HBA card type $ lspci | grep -i fibre 18:00.0 Fibre Channel: QLogic Corp. ISP2722-based 16/32Gb Fibre Channel to PCIe Adapter (rev 01) 18:00.1 Fibre Channel: QLogic Corp. ISP2722-based 16/32Gb Fibre Channel to PCIe Adapter (rev 01) Physical slot, driver, module information $ lspci -v -s

Shell, sed and awk scripting

Shell Special variables in Shell: * $0 - The filename of the current script. * $# - The number of arguments supplied to a script. * $* - All the arguments are double quoted. If a script receives two arguments, $* is equivalent to $1 $2. * $@ - All the arguments are individually double quoted. If a

Linux System Administration

Package management Install the package: $ rpm -ivh blktrace Upgrade the package: $ rpm -Uvh blktrace Remove the installed package: $ rpm -ev blktrace $ rpm -ev --nodeps blktrace Display the installed package info: $ rpm -qi blktrace Name : blktrace Relocations: (not relocatable) Version : 1.0.1 Vendor: Red Hat, Inc. Release : 6.el6 Build