Home fio benchmark on multiple files
Post
Cancel

fio benchmark on multiple files

fio directory and filename options

To run fio benchmark on multiple files or deives, we should understand the following fio options.

  • directory=str

    Prefix filenames with this directory. Used to place files in a different location than ./. You can specify a number of directories by separating the names with a ‘:’ character. These directories will be assigned equally distributed to job clones created by numjobs as long as they are using generated filenames. If specific filename(s) are set fio will use the first listed directory, and thereby matching the filename semantic (which generates a file for each clone if not specified, but lets all clones use the same file if set).

  • filename=str

    Fio normally makes up a filename based on the job name, thread number, and file number (see filename_format). If you want to share files between threads in a job or several jobs with fixed file paths, specify a filename for each of them to override the default. If the ioengine is file based, you can specify a number of files by separating the names with a ‘:’ colon. So if you wanted a job to open /dev/sda and /dev/sdb as the two working files, you would use filename=/dev/sda:/dev/sdb. This also means that whenever this option is specified, nrfiles is ignored. The size of regular files specified by this option will be size divided by number of files unless an explicit size is specified by filesize.

Run fio on single directory

The following example runs four fio jobs on single directory dir1. Four different files are laid out automatically before the benchmark.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
$ fio --name=4kwrite --ioengine=libaio --directory=dir1 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
4kwrite: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
...
fio-3.7
Starting 4 processes
4kwrite: Laying out IO file (1 file / 10240MiB)
4kwrite: Laying out IO file (1 file / 10240MiB)
4kwrite: Laying out IO file (1 file / 10240MiB)
4kwrite: Laying out IO file (1 file / 10240MiB)
bs: 4 (f=4): [W(4)][4.5%][r=0KiB/s,w=394MiB/s][r=0,w=101k IOPS][eta 01m:46s]
<...>

$ ps -ef |grep fio | grep -v grep
root     25940 27212 23 21:10 pts/1    00:00:00 fio --name=4kwrite --ioengine=libaio --directory=dir1 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     25976 25940 27 21:10 ?        00:00:01 fio --name=4kwrite --ioengine=libaio --directory=dir1 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     25977 25940 28 21:10 ?        00:00:01 fio --name=4kwrite --ioengine=libaio --directory=dir1 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     25978 25940 28 21:10 ?        00:00:01 fio --name=4kwrite --ioengine=libaio --directory=dir1 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     25979 25940 27 21:10 ?        00:00:01 fio --name=4kwrite --ioengine=libaio --directory=dir1 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting

$ lsof | egrep "dir1"
fio       25976          root    3u      REG              253,2 10737418240   23782491 /dir1/4kwrite.0.0
fio       25977          root    3u      REG              253,2 10737418240   23782492 /dir1/4kwrite.3.0
fio       25978          root    3u      REG              253,2 10737418240   23782495 /dir1/4kwrite.2.0
fio       25979          root    3u      REG              253,2 10737418240    5234528 /dir1/4kwrite.1.0

$ ls -la dir1 | grep write
-rw-r--r-- 1 root root 10737418240 Mar  1 21:11 4kwrite.0.0
-rw-r--r-- 1 root root 10737418240 Mar  1 21:11 4kwrite.1.0
-rw-r--r-- 1 root root 10737418240 Mar  1 21:11 4kwrite.2.0
-rw-r--r-- 1 root root 10737418240 Mar  1 21:11 4kwrite.3.0

Run fio on multiple directories

The following example runs four fio jobs on two directories dir1 and dir2. Two files are laid out automatically under each directory.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
$ fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting

$ ps -ef |grep fio | grep write
root     27362 27212  3 21:13 pts/1    00:00:01 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     27396 27362 29 21:13 ?        00:00:08 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     27397 27362 30 21:13 ?        00:00:08 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     27398 27362 31 21:13 ?        00:00:09 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     27399 27362 30 21:13 ?        00:00:08 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting

$ lsof | egrep "dir1|dir2"
fio       27396          root    3u      REG              253,2 10737418240   23782491 /dir1/4kwrite.0.0
fio       27397          root    3u      REG              253,2 10737418240  538334779 /dir2/4kwrite.3.0
fio       27398          root    3u      REG              253,2 10737418240   23782492 /dir1/4kwrite.2.0
fio       27399          root    3u      REG              253,2 10737418240  538334780 /dir2/4kwrite.1.0

$ ls -ltr dir*/
dir2/:
total 20971520
-rw-r--r-- 1 root root 10737418240 Mar  1 21:13 4kwrite.3.0
-rw-r--r-- 1 root root 10737418240 Mar  1 21:13 4kwrite.1.0

dir1/:
total 20971520
-rw-r--r-- 1 root root 10737418240 Mar  1 21:13 4kwrite.2.0
-rw-r--r-- 1 root root 10737418240 Mar  1 21:13 4kwrite.0.0

If the option filename is specified, only the first listed directory will be used to create files.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
$ fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting

$ ps -ef |grep fio | grep write
root     29764 27212  8 21:17 pts/1    00:00:00 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     29798 29764 33 21:17 ?        00:00:04 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     29799 29764 35 21:17 ?        00:00:04 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     29800 29764 35 21:17 ?        00:00:04 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     29801 29764 33 21:17 ?        00:00:03 fio --name=4kwrite --ioengine=libaio --directory=dir1:dir2 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting

$ lsof | egrep "dir1|dir2"
fio       29798          root    3u      REG              253,2 10737418240   23782491 /dir1/testfile
fio       29799          root    3u      REG              253,2 10737418240   23782491 /dir1/testfile
fio       29800          root    3u      REG              253,2 10737418240   23782491 /dir1/testfile
fio       29801          root    3u      REG              253,2 10737418240   23782491 /dir1/testfile

$ ls -la dir*/
dir1/:
total 10485760
drwxr-xr-x 2 root root          22 Mar  1 21:17 .
drwxr-xr-x 7 root root         225 Mar  1 20:28 ..
-rw-r--r-- 1 root root 10737418240 Mar  1 21:18 testfile

dir2/:
total 0
drwxr-xr-x 2 root root   6 Mar  1 21:16 .
drwxr-xr-x 7 root root 225 Mar  1 20:28 ..

Run multiple fio jobs on single file

The following example runs four jobs on single file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ fio --name=4kwrite --ioengine=libaio --directory=dir1 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting

$ ps -ef |grep fio | grep write
root     28819 27212  9 21:16 pts/1    00:00:00 fio --name=4kwrite --ioengine=libaio --directory=dir1 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     28884 28819 34 21:16 ?        00:00:03 fio --name=4kwrite --ioengine=libaio --directory=dir1 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     28885 28819 33 21:16 ?        00:00:02 fio --name=4kwrite --ioengine=libaio --directory=dir1 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     28886 28819 36 21:16 ?        00:00:03 fio --name=4kwrite --ioengine=libaio --directory=dir1 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting
root     28887 28819 35 21:16 ?        00:00:03 fio --name=4kwrite --ioengine=libaio --directory=dir1 --filename=testfile --blocksize=4k --readwrite=write --filesize=10G --end_fsync=1 --numjobs=4 --iodepth=128 --direct=1 --group_reporting

$ lsof | egrep "dir1"
fio       28884          root    3u      REG              253,2 10737418240   23782491 /dir1/testfile
fio       28885          root    3u      REG              253,2 10737418240   23782491 /dir1/testfile
fio       28886          root    3u      REG              253,2 10737418240   23782491 /dir1/testfile
fio       28887          root    3u      REG              253,2 10737418240   23782491 /dir1/testfile

Run fio on multiple files under different directories

In this example, there are 2 jobs to write 8 files under different directories.

1
$ fio --name=fiojob --ioengine=libaio --filename=/fiodata1/testfile:/fiodata2/testfile:/fiodata3/testfile:/fiodata4/testfile:/fiodata5/testfile:/fiodata6/testfile:/fiodata7/testfile:/fiodata8/testfile --blocksize=4k --readwrite=write --filesize=5G --end_fsync=1 --numjobs=2 --iodepth=512 --direct=1 --group_reporting --output fio.out

Reference

This post is licensed under CC BY 4.0 by the author.

fio benchmark on multiple devices

How to uninstall ceph storage cluster

Comments powered by Disqus.