Iozone - A filesystem benchmark tool
Introduction
IOzone is a filesystem benchmark tool. The benchmark generates and measures a variety of file operations.
Iozone is useful for performing a broad filesystem analysis of a vendor’s computer platform. The benchmark tests file I/O performance for the following operations:
- Read, write, re-read, re-write, read backwards, read strided, fread, fwrite, random read, pread ,mmap, aio_read, aio_write
Example for throughput benchmark
In this example, the target is to measure the filesystem throughput(KB/s) for different workloads with 4k iosize. The workload operations include sequential read/write, random read/write and mix random read/write.
We run iozone on 6 mounted filesystems and 1GB files are read and written in each filesystem.
$ /opt/iozone/bin/iozone -h
-r # record size in Kb
-s # file size in Kb
-t # Number of threads or processes to use in throughput test
-I Use VxFS VX_DIRECT, O_DIRECT,or O_DIRECTIO for all file operations
-F filenames for each process/thread in throughput test
-i # Test to run (0=write/rewrite, 1=read/re-read, 2=random-read/write
3=Read-backwards, 4=Re-write-record, 5=stride-read, 6=fwrite/re-fwrite
7=fread/Re-fread, 8=random_mix, 9=pwrite/Re-pwrite, 10=pread/Re-pread
11=pwritev/Re-pwritev, 12=preadv/Re-preadv)
[...]
$ /opt/iozone/bin/iozone -i 0 -i 1 -i 2 -i 8 -r 4k -s 1g -t 6 -I -F /testmnt1/testfile1 /testmnt2/testfile1 /testmnt3/testfile1 /testmnt4/testfile1 /testmnt5testfile1 /testmnt6/testfile1 > iozone.out
$ cat iozone.out
Iozone: Performance Test of File I/O
Version $Revision: 3.489 $
Compiled for 64 bit mode.
Build: linux-AMD64
Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
Vangel Bojaxhi, Ben England, Vikentsi Lapa,
Alexey Skidanov, Sudhir Kumar.
Run began: Mon Mar 1 12:25:11 2021
Record Size 4 kB
File size set to 1048576 kB
O_DIRECT feature enabled
Command line used: /opt/iozone/bin/iozone -i 0 -i 1 -i 2 -i 8 -r 4k -s 1g -t 6 -I -F /testmnt1/testfile1 /testmnt2/testfile1 /testmnt3/testfile1 /testmnt4testfile1 /testmnt5/testfile1 /testmnt6/testfile1
Output is in kBytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 kBytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 6 processes
Each process writes a 1048576 kByte file in 4 kByte records
Children see throughput for 6 initial writers = 151721.80 kB/sec
Parent sees throughput for 6 initial writers = 146366.56 kB/sec
Min throughput per process = 24712.40 kB/sec
Max throughput per process = 25707.35 kB/sec
Avg throughput per process = 25286.97 kB/sec
Min xfer = 1007996.00 kB
Children see throughput for 6 rewriters = 152089.88 kB/sec
Parent sees throughput for 6 rewriters = 152084.55 kB/sec
Min throughput per process = 25109.69 kB/sec
Max throughput per process = 25674.81 kB/sec
Avg throughput per process = 25348.31 kB/sec
Min xfer = 1025500.00 kB
Children see throughput for 6 readers = 7618.06 kB/sec
Parent sees throughput for 6 readers = 7618.04 kB/sec
Min throughput per process = 1268.31 kB/sec
Max throughput per process = 1270.73 kB/sec
Avg throughput per process = 1269.68 kB/sec
Min xfer = 1046580.00 kB
Children see throughput for 6 re-readers = 7629.77 kB/sec
Parent sees throughput for 6 re-readers = 7629.74 kB/sec
Min throughput per process = 1270.79 kB/sec
Max throughput per process = 1273.63 kB/sec
Avg throughput per process = 1271.63 kB/sec
Min xfer = 1046240.00 kB
Children see throughput for 6 random readers = 7605.91 kB/sec
Parent sees throughput for 6 random readers = 7605.89 kB/sec
Min throughput per process = 1266.91 kB/sec
Max throughput per process = 1268.54 kB/sec
Avg throughput per process = 1267.65 kB/sec
Min xfer = 1047228.00 kB
Children see throughput for 6 mixed workload = 79687.92 kB/sec
Parent sees throughput for 6 mixed workload = 78974.22 kB/sec
Min throughput per process = 1275.41 kB/sec
Max throughput per process = 25449.38 kB/sec
Avg throughput per process = 13281.32 kB/sec
Min xfer = 52552.00 kB
Children see throughput for 6 random writers = 146210.38 kB/sec
Parent sees throughput for 6 random writers = 143822.17 kB/sec
Min throughput per process = 24206.99 kB/sec
Max throughput per process = 24653.06 kB/sec
Avg throughput per process = 24368.40 kB/sec
Min xfer = 1029604.00 kB
iozone test complete.
iostat
In the above test, we use a flash array with single 500TB LUN created. There are 4 active paths from host to the single LUN. Six logical volumes and filesystems are created on the single LUN.
The following is a piece of iostat output for one of the four disks(paths) when the mix random read/write workload is running. The read throughput is ~940KB/s and the write throughput is 19MB/s.
To measure the maximum throughput, we need keep increasing the number of read/write threads until the throughput(KB/s) is capped. Also, the iosize has big impact on the throughput. We may test with different I/O sizes.
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdd 0.00 0.00 189.80 4255.20 759.20 17020.80 8.00 1.13 0.25 3.07 0.13 0.18 81.80
sdd 0.00 0.00 237.20 4708.20 948.80 18832.80 8.00 1.35 0.27 3.08 0.13 0.20 96.94
sdd 0.00 0.00 235.40 4898.20 941.60 19592.80 8.00 1.37 0.27 3.06 0.13 0.19 99.54
sdd 0.00 0.00 229.60 4575.60 918.40 18302.40 8.00 1.32 0.27 3.09 0.13 0.20 94.28
sdd 0.00 0.00 229.94 4822.55 919.76 19290.22 8.00 1.32 0.26 3.06 0.13 0.19 96.89
sdd 0.00 0.00 228.80 4512.40 915.20 18049.60 8.00 1.30 0.27 3.07 0.13 0.20 94.40
sdd 0.00 0.00 234.20 4810.20 936.80 19240.80 8.00 1.33 0.26 3.08 0.13 0.19 97.10
sdd 0.00 0.00 246.60 4497.40 986.40 17989.60 8.00 1.35 0.28 3.06 0.13 0.20 97.04
sdd 0.00 0.00 106.60 1665.60 426.40 6661.70 8.00 0.55 0.31 3.05 0.14 0.22 38.26