How to use iperf to measure the network bandwidth
iPerf is a tool for measurements of the maximum achievable bandwidth on IP network. The following iperf option can be used on the client side to saturate the network bandwidth if single client thread is not sufficient.
- -P, –parallel # number of parallel client threads to run
Benchmark a 100GbE network
- Single thread test
Start iperf on the server side:
$ iperf -v
iperf version 2.0.13 (21 Jan 2019) pthreads
$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 128 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.1.240 port 5001 connected with 192.168.1.245 port 44292
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 37.7 GBytes 32.4 Gbits/sec
Run iperf benchmark on the client side:
$ iperf -c 192.168.1.240
------------------------------------------------------------
Client connecting to 192.168.1.240, TCP port 5001
TCP window size: 2.44 MByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.245 port 44292 connected with 192.168.1.240 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 37.7 GBytes 32.4 Gbits/sec
With the default single thread, the achievable bandwidth is 32.4Gb/s which is far less than the available 100Gb/s bandwidth.
- Multi-threads test
We can increase the number of iperf client threads until the maximum bandwidth is hit.
$ iperf -c 192.168.1.240 -P 1
------------------------------------------------------------
Client connecting to 192.168.1.240, TCP port 5001
TCP window size: 4.00 MByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.245 port 44436 connected with 192.168.1.240 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 38.8 GBytes 33.4 Gbits/sec
$ iperf -c 192.168.1.240 -P 2
------------------------------------------------------------
Client connecting to 192.168.1.240, TCP port 5001
TCP window size: 1.13 MByte (default)
------------------------------------------------------------
[ 4] local 192.168.1.245 port 44440 connected with 192.168.1.240 port 5001
[ 3] local 192.168.1.245 port 44438 connected with 192.168.1.240 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 35.3 GBytes 30.4 Gbits/sec
[ 3] 0.0-10.0 sec 37.6 GBytes 32.3 Gbits/sec
[SUM] 0.0-10.0 sec 72.9 GBytes 62.6 Gbits/sec
$ iperf -c 192.168.1.240 -P 4
------------------------------------------------------------
Client connecting to 192.168.1.240, TCP port 5001
TCP window size: 366 KByte (default)
------------------------------------------------------------
[ 6] local 192.168.1.245 port 44448 connected with 192.168.1.240 port 5001
[ 5] local 192.168.1.245 port 44446 connected with 192.168.1.240 port 5001
[ 4] local 192.168.1.245 port 44444 connected with 192.168.1.240 port 5001
[ 3] local 192.168.1.245 port 44442 connected with 192.168.1.240 port 5001
[ ID] Interval Transfer Bandwidth
[ 6] 0.0-10.0 sec 26.0 GBytes 22.3 Gbits/sec
[ 5] 0.0-10.0 sec 26.3 GBytes 22.6 Gbits/sec
[ 4] 0.0-10.0 sec 26.3 GBytes 22.6 Gbits/sec
[ 3] 0.0-10.0 sec 26.4 GBytes 22.6 Gbits/sec
[SUM] 0.0-10.0 sec 105 GBytes 90.1 Gbits/sec
$ iperf -c 192.168.1.240 -P 8
------------------------------------------------------------
Client connecting to 192.168.1.240, TCP port 5001
TCP window size: 518 KByte (default)
------------------------------------------------------------
[ 16] local 192.168.1.245 port 44464 connected with 192.168.1.240 port 5001
[ 3] local 192.168.1.245 port 44452 connected with 192.168.1.240 port 5001
[ 7] local 192.168.1.245 port 44456 connected with 192.168.1.240 port 5001
[ 4] local 192.168.1.245 port 44450 connected with 192.168.1.240 port 5001
[ 6] local 192.168.1.245 port 44454 connected with 192.168.1.240 port 5001
[ 9] local 192.168.1.245 port 44462 connected with 192.168.1.240 port 5001
[ 8] local 192.168.1.245 port 44460 connected with 192.168.1.240 port 5001
[ 5] local 192.168.1.245 port 44458 connected with 192.168.1.240 port 5001
[ ID] Interval Transfer Bandwidth
[ 16] 0.0-10.0 sec 15.0 GBytes 12.9 Gbits/sec
[ 3] 0.0-10.0 sec 9.06 GBytes 7.78 Gbits/sec
[ 7] 0.0-10.0 sec 15.1 GBytes 12.9 Gbits/sec
[ 4] 0.0-10.0 sec 15.1 GBytes 13.0 Gbits/sec
[ 6] 0.0-10.0 sec 15.1 GBytes 12.9 Gbits/sec
[ 9] 0.0-10.0 sec 15.1 GBytes 12.9 Gbits/sec
[ 8] 0.0-10.0 sec 8.97 GBytes 7.70 Gbits/sec
[ 5] 0.0-10.0 sec 15.1 GBytes 12.9 Gbits/sec
[SUM] 0.0-10.0 sec 108 GBytes 93.1 Gbits/sec
We can verify the network throughput with the command sar -n DEV 2.
03:26:05 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
03:26:07 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
03:26:07 PM eth0 15.00 0.50 0.91 0.33 0.00 0.00 0.00
03:26:07 PM eth3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
03:26:07 PM eth1 8045242.00 252491.50 11895015.32 16273.87 0.00 0.00 0.00
03:26:07 PM eth2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
03:26:07 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
03:26:10 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
03:26:10 PM eth0 10.50 1.00 0.65 0.41 0.00 0.00 0.00
03:26:10 PM eth3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
03:26:10 PM eth1 8046641.50 251968.50 11897084.51 16240.16 0.00 0.00 0.00
03:26:10 PM eth2 0.00 0.00 0.00 0.00 0.00 0.00 0.00