libaio init failed due to resource temporarily unavailable
Issue Description
$ sudo fio --blocksize=64k --directory=/mnt/bench1 --filename=testfile --ioengine=libaio --readwrite=randread --size=10G --name=test --numjobs=512 --group_reporting --direct=1 --iodepth=128 --randrepeat=1 --disable_lat=0 --gtod_reduce=0
test: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=128
...
fio-3.7
Starting 512 processes
fio: pid=38868, err=11/file:engines/libaio.c:354, func=io_queue_init, error=Resource temporarily unavailable
...
fio: check /proc/sys/fs/aio-max-nr
fio: io engine libaio init failed. Perhaps try reducing io depth?
Resolution
The Linux kernel provides the Asynchronous non-blocking I/O (AIO) feature that allows a process to initiate multiple I/O operations simultaneously without having to wait for any of them to complete. This helps boost performance for applications that are able to overlap processing and I/O.
The performance can be tuned using the /proc/sys/fs/aio-max-nr virtual file in the proc file system. The aio-max-nr parameter determines the maximum number of allowable concurrent requests.
To set the aio-max-nr value, add the following line to the /etc/sysctl.d/99-sysctl.conf file:
$ cat /proc/sys/fs/aio-max-nr
65536
$ echo "fs.aio-max-nr = 1048576" >> /etc/sysctl.d/99-sysctl.conf
To activate the new setting, run the following command:
$ sysctl -p /etc/sysctl.d/99-sysctl.conf
fs.aio-max-nr = 1048576