fio benchmark on multiple files
fio directory and filename options
To run fio benchmark on multiple files or deives, we should understand the following fio options.
- directory=str
Prefix filenames with this directory. Used to place files in a different location than ./. You can specify a number of directories by separating the names with a ‘:’ character. These directories will be assigned equally distributed to job clones created by numjobs as long as they are using generated filenames. If specific filename(s) are set fio will use the first listed directory, and thereby matching the filename semantic (which generates a file for each clone if not specified, but lets all clones use the same file if set).
- filename=str
Fio normally makes up a filename based on the job name, thread number, and file number (see filename_format). If you want to share files between threads in a job or several jobs with fixed file paths, specify a filename for each of them to override the default. If the ioengine is file based, you can specify a number of files by separating the names with a ‘:’ colon. So if you wanted a job to open /dev/sda and /dev/sdb as the two working files, you would use filename=/dev/sda:/dev/sdb. This also means that whenever this option is specified, nrfiles is ignored. The size of regular files specified by this option will be size divided by number of files unless an explicit size is specified by filesize.
Run fio on single directory
The following example runs four fio jobs on single directory dir1. Four different files are laid out automatically before the benchmark.
$ fio --name=4kwrite --ioengine=libaio --directory=dir1 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
4kwrite: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
...
fio-3.7
Starting 4 processes
4kwrite: Laying out IO file (1 file / 10240MiB)
4kwrite: Laying out IO file (1 file / 10240MiB)
4kwrite: Laying out IO file (1 file / 10240MiB)
4kwrite: Laying out IO file (1 file / 10240MiB)
bs: 4 (f=4): [W(4)][4.5%][r=0KiB/s,w=394MiB/s][r=0,w=101k IOPS][eta 01m:46s]
<...>
$ ps -ef |grep fio | grep -v grep
root 25940 27212 23 21:10 pts/1 00:00:00 fio --name=4kwrite --ioengine=libaio --directory=dir1 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root 25976 25940 27 21:10 ? 00:00:01 fio --name=4kwrite --ioengine=libaio --directory=dir1 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root 25977 25940 28 21:10 ? 00:00:01 fio --name=4kwrite --ioengine=libaio --directory=dir1 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root 25978 25940 28 21:10 ? 00:00:01 fio --name=4kwrite --ioengine=libaio --directory=dir1 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root 25979 25940 27 21:10 ? 00:00:01 fio --name=4kwrite --ioengine=libaio --directory=dir1 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
$ lsof | egrep "dir1"
fio 25976 root 3u REG 253,2 10737418240 23782491 /dir1/4kwrite.0.0
fio 25977 root 3u REG 253,2 10737418240 23782492 /dir1/4kwrite.3.0
fio 25978 root 3u REG 253,2 10737418240 23782495 /dir1/4kwrite.2.0
fio 25979 root 3u REG 253,2 10737418240 5234528 /dir1/4kwrite.1.0
$ ls -la dir1 | grep write
-rw-r--r-- 1 root root 10737418240 Mar 1 21:11 4kwrite.0.0
-rw-r--r-- 1 root root 10737418240 Mar 1 21:11 4kwrite.1.0
-rw-r--r-- 1 root root 10737418240 Mar 1 21:11 4kwrite.2.0
-rw-r--r-- 1 root root 10737418240 Mar 1 21:11 4kwrite.3.0
Run fio on multiple directories
The following example runs four fio jobs on two directories dir1 and dir2. Two files are laid out automatically under each directory.
$ fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
$ ps -ef |grep fio | grep write
root 27362 27212 3 21:13 pts/1 00:00:01 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root 27396 27362 29 21:13 ? 00:00:08 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root 27397 27362 30 21:13 ? 00:00:08 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root 27398 27362 31 21:13 ? 00:00:09 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root 27399 27362 30 21:13 ? 00:00:08 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
$ lsof | egrep "dir1|dir2"
fio 27396 root 3u REG 253,2 10737418240 23782491 /dir1/4kwrite.0.0
fio 27397 root 3u REG 253,2 10737418240 538334779 /dir2/4kwrite.3.0
fio 27398 root 3u REG 253,2 10737418240 23782492 /dir1/4kwrite.2.0
fio 27399 root 3u REG 253,2 10737418240 538334780 /dir2/4kwrite.1.0
$ ls -ltr dir*/
dir2/:
total 20971520
-rw-r--r-- 1 root root 10737418240 Mar 1 21:13 4kwrite.3.0
-rw-r--r-- 1 root root 10737418240 Mar 1 21:13 4kwrite.1.0
dir1/:
total 20971520
-rw-r--r-- 1 root root 10737418240 Mar 1 21:13 4kwrite.2.0
-rw-r--r-- 1 root root 10737418240 Mar 1 21:13 4kwrite.0.0
If the option filename is specified, only the first listed directory will be used to create files.
$ fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
$ ps -ef |grep fio | grep write
root 29764 27212 8 21:17 pts/1 00:00:00 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root 29798 29764 33 21:17 ? 00:00:04 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root 29799 29764 35 21:17 ? 00:00:04 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root 29800 29764 35 21:17 ? 00:00:04 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root 29801 29764 33 21:17 ? 00:00:03 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
$ lsof | egrep "dir1|dir2"
fio 29798 root 3u REG 253,2 10737418240 23782491 /dir1/testfile
fio 29799 root 3u REG 253,2 10737418240 23782491 /dir1/testfile
fio 29800 root 3u REG 253,2 10737418240 23782491 /dir1/testfile
fio 29801 root 3u REG 253,2 10737418240 23782491 /dir1/testfile
$ ls -la dir*/
dir1/:
total 10485760
drwxr-xr-x 2 root root 22 Mar 1 21:17 .
drwxr-xr-x 7 root root 225 Mar 1 20:28 ..
-rw-r--r-- 1 root root 10737418240 Mar 1 21:18 testfile
dir2/:
total 0
drwxr-xr-x 2 root root 6 Mar 1 21:16 .
drwxr-xr-x 7 root root 225 Mar 1 20:28 ..
Run multiple fio jobs on single file
The following example runs four jobs on single file.
$ fio --name=4kwrite --ioengine=libaio --directory=dir1 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
$ ps -ef |grep fio | grep write
root 28819 27212 9 21:16 pts/1 00:00:00 fio --name=4kwrite --ioengine=libaio --directory=dir1 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root 28884 28819 34 21:16 ? 00:00:03 fio --name=4kwrite --ioengine=libaio --directory=dir1 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root 28885 28819 33 21:16 ? 00:00:02 fio --name=4kwrite --ioengine=libaio --directory=dir1 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root 28886 28819 36 21:16 ? 00:00:03 fio --name=4kwrite --ioengine=libaio --directory=dir1 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root 28887 28819 35 21:16 ? 00:00:03 fio --name=4kwrite --ioengine=libaio --directory=dir1 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
$ lsof | egrep "dir1"
fio 28884 root 3u REG 253,2 10737418240 23782491 /dir1/testfile
fio 28885 root 3u REG 253,2 10737418240 23782491 /dir1/testfile
fio 28886 root 3u REG 253,2 10737418240 23782491 /dir1/testfile
fio 28887 root 3u REG 253,2 10737418240 23782491 /dir1/testfile
Run fio on multiple files from different directories
One job writes two files
In this example, there is one fio job to write two files from two different directories. The total iodepth on the two files is 128. Note that, the iodepth for each file is about 64 which is only half of the specified iodepth in the fio command.
$ fio --blocksize=4k --filename=/mnt/dir1/testfile:/mnt/dir2/testfile --ioengine=libaio --readwrite=write --size=50G --name=test --numjobs=1 --group_reporting --direct=1 --iodepth=128 --end_fsync=1
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.7
Starting 1 process
<...>
$ lsof | egrep "/mnt/dir"
fio 74145 root 3u REG 252,1 53687091200 11 /mnt/dir1/testfile
fio 74145 root 4u REG 252,2 53687091200 11 /mnt/dir2/testfile
$ iostat -ktdx 2
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
pxd!pxd90182615933154185 0.00 0.00 0.00 98857.00 0.00 395428.00 8.00 62.86 0.64 0.00 0.64 0.01 100.00
pxd!pxd798820514973607815 0.00 0.00 0.00 98858.00 0.00 395432.00 8.00 62.84 0.64 0.00 0.64 0.01 100.00
Three jobs write two files
In this example, there are three fio jobs and each job is to write two files. The actual iodepth on each file is ~184(roughly = 128/2 * 3) which is the accumulated iodepth from three jobs.
$ fio --blocksize=4k --filename=/mnt/dir1/testfile:/mnt/dir2/testfile --ioengine=libaio --readwrite=write --size=50G --name=test --numjobs=3 --group_reporting --direct=1 --iodepth=128 --end_fsync=1
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
...
fio-3.7
Starting 3 processes
<...>
$ lsof | egrep "/mnt/dir"
fio 85081 root 3u REG 252,1 53687091200 11 /mnt/dir1/testfile
fio 85081 root 4u REG 252,2 53687091200 11 /mnt/dir2/testfile
fio 85082 root 3u REG 252,1 53687091200 11 /mnt/dir1/testfile
fio 85082 root 4u REG 252,2 53687091200 11 /mnt/dir2/testfile
fio 85083 root 3u REG 252,1 53687091200 11 /mnt/dir1/testfile
fio 85083 root 4u REG 252,2 53687091200 11 /mnt/dir2/testfile
$ iostat -ktex 2
pxd!pxd90182615933154185 0.00 0.50 0.00 99324.50 0.00 397300.00 8.00 184.13 1.85 0.00 1.85 0.01 100.00
pxd!pxd798820514973607815 0.00 0.50 0.00 99324.50 0.00 397300.00 8.00 184.02 1.85 0.00 1.85 0.01 100.00
Using dedicated jobs writes each file
In this example, there are two fio jobs and each job is to write a different file. The actual iodepth on each file is ~128 which is the same as the specified iodepth in the fio command. This is usually expected pattern in the benchmark.
$ fio --blocksize=4k --ioengine=libaio --readwrite=write --size=50G --direct=1 --iodepth=128 --end_fsync=1 --group_reporting --numjobs=1 --name=job1 --filename=/mnt/dir1/testfile --name=job2 --filename=/mnt/dir2/testfile
job1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
job2: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.7
Starting 2 processes
<...>
$ lsof | egrep "/mnt/dir"
fio 79794 root 3u REG 252,1 53687091200 11 /mnt/dir1/testfile
fio 79795 root 3u REG 252,2 53687091200 11 /mnt/dir2/testfile
$ iostat -ktdx 2
pxd!pxd90182615933154185 0.00 0.00 0.00 94151.00 0.00 376604.00 8.00 127.01 1.35 0.00 1.35 0.01 100.00
pxd!pxd798820514973607815 0.00 0.00 0.00 94152.50 0.00 376610.00 8.00 127.01 1.35 0.00 1.35 0.01 100.00
In this example, there are four fio jobs and each file is written by two jobs. The actual iodepth on each file is ~256 which is the twice of the specified iodepth in the fio command.
$ fio --blocksize=4k --ioengine=libaio --readwrite=write --size=50G --direct=1 --iodepth=128 --end_fsync=1 --group_reporting --numjobs=2 --name=job1 --filename=/mnt/dir1/testfile --name=job2 --filename=/mnt/dir2/testfile
job1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
...
job2: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
...
fio-3.7
Starting 4 processes
<...>
$ lsof | egrep "/mnt/dir"
fio 81972 root 3u REG 252,1 53687091200 11 /mnt/dir1/testfile
fio 81973 root 3u REG 252,1 53687091200 11 /mnt/dir1/testfile
fio 81974 root 3u REG 252,2 53687091200 11 /mnt/dir2/testfile
fio 81975 root 3u REG 252,2 53687091200 11 /mnt/dir2/testfile
$ iostat -ktdx 2
pxd!pxd90182615933154185 0.00 0.50 0.00 93394.50 0.00 373580.00 8.00 254.94 2.73 0.00 2.73 0.01 100.00
pxd!pxd798820514973607815 0.00 0.50 0.00 93408.00 0.00 373634.00 8.00 254.96 2.73 0.00 2.73 0.01 100.00