How to run fio across multiple servers

Posted on Edited on In , Views:

Client-server mode

Normally fio is invoked as a stand-alone application on the machine where the I/O workload should be generated. However, the backend and frontend of fio can be run separately i.e., the fio server can generate an I/O workload on the “Device Under Test” while being controlled by a client on another machine.

Start the server on the machine which has access to the storage DUT.

  • Start a fio server, listening on all interfaces on the default port (8765).

    1
    $ fio --server

  • Start a fio server, listening on IP belonging to hostname and on specified port.

    1
    $ fio --server=<ip>:<hostname>,<port>

Example

Start server on vm1 and vm2 and listen on default port 8765:

1
2
3
4
5
[root@vm1 ~]# fio -server
fio: server listening on 0.0.0.0,8765

[root@vm2 ~]# fio -server
fio: server listening on 0.0.0.0,8765

If you have multiple servers, you can input a file containing host IPs/names for the –client option.

1
2
3
[root@vm3 ~]# cat host.list.2vms
10.10.10.2
10.10.10.3

Create a fio job file to be run on each server:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[root@vm3 ~]#  cat fio.job
[global]
ioengine=libaio
size=5GiB
name=test
direct=1
iodepth=128
group_reporting=1
numjobs=2

[job1]
blocksize=64k
filename=/dev/sdd
readwrite=write
end_fsync=1

Trigger fio jobs from client and run the jobs across multiple servers:

1
[root@vm3 ~]# fio --client=./host.list.2vms --output=fio.2vms.out fio.job

Check the fio result:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
[root@vm3 ~]# cat fio.2vms.out
hostname=vm1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-3.7, flags=1
hostname=vm2, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-3.7, flags=1
<vm1> job1: (g=0): rw=write, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=128
<vm1> ...
<vm2> job1: (g=0): rw=write, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=128
<vm2> ...


<vm1> Starting 2 processes
<vm2> Starting 2 processes


job1: (groupid=0, jobs=2): err= 0: pid=25680: Sat Sep 23 04:41:08 2023
  write: IOPS=2864, BW=179MiB/s (188MB/s)(9537MiB/53276msec)
    slat (usec): min=2, max=302, avg=11.76, stdev= 7.74
    clat (msec): min=3, max=191, avg=89.36, stdev=15.47
     lat (msec): min=3, max=191, avg=89.37, stdev=15.47
    clat percentiles (msec):
     |  1.00th=[   59],  5.00th=[   62], 10.00th=[   64], 20.00th=[   70],
     | 30.00th=[   87], 40.00th=[   92], 50.00th=[   94], 60.00th=[   96],
     | 70.00th=[   99], 80.00th=[  103], 90.00th=[  106], 95.00th=[  109],
     | 99.00th=[  114], 99.50th=[  116], 99.90th=[  129], 99.95th=[  163],
     | 99.99th=[  192]
   bw (  KiB/s): min=79744, max=129664, per=50.01%, avg=91671.96, stdev=15600.42, samples=212
   iops        : min= 1246, max= 2026, avg=1432.36, stdev=243.76, samples=212
  lat (msec)   : 4=0.02%, 10=0.03%, 20=0.03%, 50=0.08%, 100=74.58%
  lat (msec)   : 250=25.26%
  cpu          : usr=0.67%, sys=1.32%, ctx=9877, majf=0, minf=19
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,152588,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
  WRITE: bw=179MiB/s (188MB/s), 179MiB/s-179MiB/s (188MB/s-188MB/s), io=9537MiB (10.0GB), run=53276-53276msec

Disk stats (read/write):
  sdd: ios=48/151321, merge=0/1119, ticks=34/13488966, in_queue=13489000, util=99.84%
<vm1>


job1: (groupid=0, jobs=2): err= 0: pid=4425: Sat Sep 23 04:41:08 2023
  write: IOPS=2855, BW=178MiB/s (187MB/s)(9537MiB/53430msec)
    slat (usec): min=2, max=269, avg=12.22, stdev= 9.09
    clat (msec): min=4, max=137, avg=89.61, stdev=15.68
     lat (msec): min=4, max=137, avg=89.63, stdev=15.68
    clat percentiles (msec):
     |  1.00th=[   56],  5.00th=[   62], 10.00th=[   63], 20.00th=[   70],
     | 30.00th=[   87], 40.00th=[   92], 50.00th=[   94], 60.00th=[   97],
     | 70.00th=[  100], 80.00th=[  103], 90.00th=[  106], 95.00th=[  110],
     | 99.00th=[  115], 99.50th=[  117], 99.90th=[  123], 99.95th=[  125],
     | 99.99th=[  136]
   bw (  KiB/s): min=77952, max=128896, per=49.89%, avg=91190.65, stdev=15736.83, samples=212
   iops        : min= 1218, max= 2014, avg=1424.84, stdev=245.89, samples=212
  lat (msec)   : 10=0.02%, 20=0.03%, 50=0.22%, 100=71.90%, 250=27.82%
  cpu          : usr=0.66%, sys=1.37%, ctx=9653, majf=0, minf=19
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,152588,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128


All clients: (groupid=0, jobs=2): err= 0: pid=0: Sat Sep 23 04:41:08 2023
  write: IOPS=5711, BW=357Mi (374M)(18.6GiB/53430msec)
    slat (usec): min=2, max=302, avg=12.22, stdev= 6.43
    clat (msec): min=3, max=191, avg=89.61, stdev=11.09
     lat (msec): min=3, max=191, avg=89.63, stdev=11.09
   bw (  KiB/s): min=77952, max=129664, per=24.91%, avg=91190.65, stdev=11114.46, samples=424
   iops        : min= 1218, max= 2026, avg=1424.84, stdev=173.67, samples=424
  lat (msec)   : 4=0.01%, 10=0.02%, 20=0.03%, 50=0.15%, 100=73.24%
  lat (msec)   : 250=26.54%
  cpu          : usr=0.66%, sys=1.35%, ctx=19530, majf=0, minf=38
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,305176,0,0 short=0,0,0,0 dropped=0,0,0,0

Run status group 0 (all jobs):
  WRITE: bw=178MiB/s (187MB/s), 178MiB/s-178MiB/s (187MB/s-187MB/s), io=9537MiB (10.0GB), run=53430-53430msec


Disk stats (read/write):
  sdd: ios=84/151410, merge=0/1068, ticks=30/13539292, in_queue=13539322, util=99.77%
<vm2>