Home fio benchmark on multiple files
Post
Cancel

fio benchmark on multiple files

fio directory and filename options

To run fio benchmark on multiple files or deives, we should understand the following fio options.

  • directory=str

    Prefix filenames with this directory. Used to place files in a different location than ./. You can specify a number of directories by separating the names with a ‘:’ character. These directories will be assigned equally distributed to job clones created by numjobs as long as they are using generated filenames. If specific filename(s) are set fio will use the first listed directory, and thereby matching the filename semantic (which generates a file for each clone if not specified, but lets all clones use the same file if set).

  • filename=str

    Fio normally makes up a filename based on the job name, thread number, and file number (see filename_format). If you want to share files between threads in a job or several jobs with fixed file paths, specify a filename for each of them to override the default. If the ioengine is file based, you can specify a number of files by separating the names with a ‘:’ colon. So if you wanted a job to open /dev/sda and /dev/sdb as the two working files, you would use filename=/dev/sda:/dev/sdb. This also means that whenever this option is specified, nrfiles is ignored. The size of regular files specified by this option will be size divided by number of files unless an explicit size is specified by filesize.

Run fio on single directory

The following example runs four fio jobs on single directory dir1. Four different files are laid out automatically before the benchmark.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
$ fio --name=4kwrite --ioengine=libaio --directory=dir1 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
4kwrite: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
...
fio-3.7
Starting 4 processes
4kwrite: Laying out IO file (1 file / 10240MiB)
4kwrite: Laying out IO file (1 file / 10240MiB)
4kwrite: Laying out IO file (1 file / 10240MiB)
4kwrite: Laying out IO file (1 file / 10240MiB)
bs: 4 (f=4): [W(4)][4.5%][r=0KiB/s,w=394MiB/s][r=0,w=101k IOPS][eta 01m:46s]
<...>

$ ps -ef |grep fio | grep -v grep
root     25940 27212 23 21:10 pts/1    00:00:00 fio --name=4kwrite --ioengine=libaio --directory=dir1 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     25976 25940 27 21:10 ?        00:00:01 fio --name=4kwrite --ioengine=libaio --directory=dir1 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     25977 25940 28 21:10 ?        00:00:01 fio --name=4kwrite --ioengine=libaio --directory=dir1 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     25978 25940 28 21:10 ?        00:00:01 fio --name=4kwrite --ioengine=libaio --directory=dir1 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     25979 25940 27 21:10 ?        00:00:01 fio --name=4kwrite --ioengine=libaio --directory=dir1 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting

$ lsof | egrep "dir1"
fio       25976          root    3u      REG              253,2 10737418240   23782491 /dir1/4kwrite.0.0
fio       25977          root    3u      REG              253,2 10737418240   23782492 /dir1/4kwrite.3.0
fio       25978          root    3u      REG              253,2 10737418240   23782495 /dir1/4kwrite.2.0
fio       25979          root    3u      REG              253,2 10737418240    5234528 /dir1/4kwrite.1.0

$ ls -la dir1 | grep write
-rw-r--r-- 1 root root 10737418240 Mar  1 21:11 4kwrite.0.0
-rw-r--r-- 1 root root 10737418240 Mar  1 21:11 4kwrite.1.0
-rw-r--r-- 1 root root 10737418240 Mar  1 21:11 4kwrite.2.0
-rw-r--r-- 1 root root 10737418240 Mar  1 21:11 4kwrite.3.0

Run fio on multiple directories

The following example runs four fio jobs on two directories dir1 and dir2. Two files are laid out automatically under each directory.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
$ fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting

$ ps -ef |grep fio | grep write
root     27362 27212  3 21:13 pts/1    00:00:01 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     27396 27362 29 21:13 ?        00:00:08 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     27397 27362 30 21:13 ?        00:00:08 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     27398 27362 31 21:13 ?        00:00:09 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     27399 27362 30 21:13 ?        00:00:08 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting

$ lsof | egrep "dir1|dir2"
fio       27396          root    3u      REG              253,2 10737418240   23782491 /dir1/4kwrite.0.0
fio       27397          root    3u      REG              253,2 10737418240  538334779 /dir2/4kwrite.3.0
fio       27398          root    3u      REG              253,2 10737418240   23782492 /dir1/4kwrite.2.0
fio       27399          root    3u      REG              253,2 10737418240  538334780 /dir2/4kwrite.1.0

$ ls -ltr dir*/
dir2/:
total 20971520
-rw-r--r-- 1 root root 10737418240 Mar  1 21:13 4kwrite.3.0
-rw-r--r-- 1 root root 10737418240 Mar  1 21:13 4kwrite.1.0

dir1/:
total 20971520
-rw-r--r-- 1 root root 10737418240 Mar  1 21:13 4kwrite.2.0
-rw-r--r-- 1 root root 10737418240 Mar  1 21:13 4kwrite.0.0

If the option filename is specified, only the first listed directory will be used to create files.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
$ fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting

$ ps -ef |grep fio | grep write
root     29764 27212  8 21:17 pts/1    00:00:00 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     29798 29764 33 21:17 ?        00:00:04 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     29799 29764 35 21:17 ?        00:00:04 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     29800 29764 35 21:17 ?        00:00:04 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     29801 29764 33 21:17 ?        00:00:03 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting

$ lsof | egrep "dir1|dir2"
fio       29798          root    3u      REG              253,2 10737418240   23782491 /dir1/testfile
fio       29799          root    3u      REG              253,2 10737418240   23782491 /dir1/testfile
fio       29800          root    3u      REG              253,2 10737418240   23782491 /dir1/testfile
fio       29801          root    3u      REG              253,2 10737418240   23782491 /dir1/testfile

$ ls -la dir*/
dir1/:
total 10485760
drwxr-xr-x 2 root root          22 Mar  1 21:17 .
drwxr-xr-x 7 root root         225 Mar  1 20:28 ..
-rw-r--r-- 1 root root 10737418240 Mar  1 21:18 testfile

dir2/:
total 0
drwxr-xr-x 2 root root   6 Mar  1 21:16 .
drwxr-xr-x 7 root root 225 Mar  1 20:28 ..

Run multiple fio jobs on single file

The following example runs four jobs on single file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ fio --name=4kwrite --ioengine=libaio --directory=dir1 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting

$ ps -ef |grep fio | grep write
root     28819 27212  9 21:16 pts/1    00:00:00 fio --name=4kwrite --ioengine=libaio --directory=dir1 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     28884 28819 34 21:16 ?        00:00:03 fio --name=4kwrite --ioengine=libaio --directory=dir1 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     28885 28819 33 21:16 ?        00:00:02 fio --name=4kwrite --ioengine=libaio --directory=dir1 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     28886 28819 36 21:16 ?        00:00:03 fio --name=4kwrite --ioengine=libaio --directory=dir1 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     28887 28819 35 21:16 ?        00:00:03 fio --name=4kwrite --ioengine=libaio --directory=dir1 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting

$ lsof | egrep "dir1"
fio       28884          root    3u      REG              253,2 10737418240   23782491 /dir1/testfile
fio       28885          root    3u      REG              253,2 10737418240   23782491 /dir1/testfile
fio       28886          root    3u      REG              253,2 10737418240   23782491 /dir1/testfile
fio       28887          root    3u      REG              253,2 10737418240   23782491 /dir1/testfile

Run fio on multiple files from different directories

One job writes two files

In this example, there is one fio job to write two files from two different directories. The total iodepth on the two files is 128. Note that, the iodepth for each file is about 64 which is only half of the specified iodepth in the fio command.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ fio --blocksize=4k --filename=/mnt/dir1/testfile:/mnt/dir2/testfile --ioengine=libaio --readwrite=write --size=50G --name=test --numjobs=1 --group_reporting --direct=1 --iodepth=128 --end_fsync=1
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.7
Starting 1 process
<...>

$ lsof | egrep "/mnt/dir"
fio       74145                 root    3u      REG              252,1 53687091200         11 /mnt/dir1/testfile
fio       74145                 root    4u      REG              252,2 53687091200         11 /mnt/dir2/testfile

$ iostat -ktdx 2
Device:                     rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
pxd!pxd90182615933154185     0.00     0.00    0.00 98857.00     0.00 395428.00     8.00    62.86    0.64    0.00    0.64   0.01 100.00
pxd!pxd798820514973607815     0.00     0.00    0.00 98858.00     0.00 395432.00     8.00    62.84    0.64    0.00    0.64   0.01 100.00

Three jobs write two files

In this example, there are three fio jobs and each job is to write two files. The actual iodepth on each file is ~184(roughly = 128/2 * 3) which is the accumulated iodepth from three jobs.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ fio --blocksize=4k --filename=/mnt/dir1/testfile:/mnt/dir2/testfile --ioengine=libaio --readwrite=write --size=50G --name=test --numjobs=3 --group_reporting --direct=1 --iodepth=128 --end_fsync=1
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
...
fio-3.7
Starting 3 processes
<...>

$ lsof | egrep "/mnt/dir"
fio       85081                 root    3u      REG              252,1 53687091200         11 /mnt/dir1/testfile
fio       85081                 root    4u      REG              252,2 53687091200         11 /mnt/dir2/testfile
fio       85082                 root    3u      REG              252,1 53687091200         11 /mnt/dir1/testfile
fio       85082                 root    4u      REG              252,2 53687091200         11 /mnt/dir2/testfile
fio       85083                 root    3u      REG              252,1 53687091200         11 /mnt/dir1/testfile
fio       85083                 root    4u      REG              252,2 53687091200         11 /mnt/dir2/testfile

$ iostat -ktex 2 
pxd!pxd90182615933154185     0.00     0.50    0.00 99324.50     0.00 397300.00     8.00   184.13    1.85    0.00    1.85   0.01 100.00
pxd!pxd798820514973607815     0.00     0.50    0.00 99324.50     0.00 397300.00     8.00   184.02    1.85    0.00    1.85   0.01 100.00

Using dedicated jobs writes each file

In this example, there are two fio jobs and each job is to write a different file. The actual iodepth on each file is ~128 which is the same as the specified iodepth in the fio command. This is usually expected pattern in the benchmark.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ fio --blocksize=4k --ioengine=libaio --readwrite=write --size=50G --direct=1 --iodepth=128 --end_fsync=1 --group_reporting --numjobs=1 --name=job1 --filename=/mnt/dir1/testfile --name=job2 --filename=/mnt/dir2/testfile
job1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
job2: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.7
Starting 2 processes
<...>

$ lsof | egrep "/mnt/dir"
fio       79794                 root    3u      REG              252,1 53687091200         11 /mnt/dir1/testfile
fio       79795                 root    3u      REG              252,2 53687091200         11 /mnt/dir2/testfile

$ iostat -ktdx 2
pxd!pxd90182615933154185     0.00     0.00    0.00 94151.00     0.00 376604.00     8.00   127.01    1.35    0.00    1.35   0.01 100.00
pxd!pxd798820514973607815     0.00     0.00    0.00 94152.50     0.00 376610.00     8.00   127.01    1.35    0.00    1.35   0.01 100.00

In this example, there are four fio jobs and each file is written by two jobs. The actual iodepth on each file is ~256 which is the twice of the specified iodepth in the fio command.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ fio --blocksize=4k --ioengine=libaio --readwrite=write --size=50G --direct=1 --iodepth=128 --end_fsync=1 --group_reporting --numjobs=2 --name=job1 --filename=/mnt/dir1/testfile --name=job2 --filename=/mnt/dir2/testfile
job1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
...
job2: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
...
fio-3.7
Starting 4 processes
<...>

$ lsof | egrep "/mnt/dir"
fio       81972                 root    3u      REG              252,1 53687091200         11 /mnt/dir1/testfile
fio       81973                 root    3u      REG              252,1 53687091200         11 /mnt/dir1/testfile
fio       81974                 root    3u      REG              252,2 53687091200         11 /mnt/dir2/testfile
fio       81975                 root    3u      REG              252,2 53687091200         11 /mnt/dir2/testfile

$ iostat -ktdx 2
pxd!pxd90182615933154185     0.00     0.50    0.00 93394.50     0.00 373580.00     8.00   254.94    2.73    0.00    2.73   0.01 100.00
pxd!pxd798820514973607815     0.00     0.50    0.00 93408.00     0.00 373634.00     8.00   254.96    2.73    0.00    2.73   0.01 100.00

Reference

This post is licensed under CC BY 4.0 by the author.

fio benchmark on multiple devices

How to uninstall ceph storage cluster

Comments powered by Disqus.