Source:
https://i0.wp.com/www.virtono.com/community/wp-content/uploads/2018/02/handson-lab-how-to-unleash-your-storage-performance-by-using-nvm-express-based-pci-express-solidstate-drives-30-638.jpg?fit=638%2C359&ssl=1
https://community.mellanox.com/s/article/howto-install-flexible-i-o–fio–for-storage-benchmarking-i-o-testing
This post discusses the download, compilation, and use of Flexible I/O (fio) package for I/O benchmarking.
References
- Performance Tuning for Mellanox Adapters
- GitHub – Flexible I/O Tester
- Flexible I/O tester – Linux man page
Configuration
1. Download the latest fio package.
2. Unzip, and open the package.
# gunzip fio-2.1.10.tar.gz
# tar -xf fio-2.1.10.tar
3. Configure, run make and make install.
# cd fio-2.1.10/
# ./configure
# make
# make install
4. Run fio -h.
# fio -h
fio-2.1.10
fio [options] [job options] <job file(s)>
–debug=options Enable debug logging. May be one/more of:
process,file,io,mem,blktrace,verify,random,parse,
diskutil,job,mutex,profile,time,net,rate
–parse-only Parse options only, don’t start any IO
–output Write output to file
–runtime Runtime in seconds
–latency-log Generate per-job latency logs
–bandwidth-log Generate per-job bandwidth logs
–minimal Minimal (terse) output
–output-format=x Output format (terse,json,normal)
–terse-version=x Set terse version output format to ‚x‘
–version Print version info and exit
–help Print this page
–cpuclock-test Perform test/validation of CPU clock
–crctest Test speed of checksum functions
–cmdhelp=cmd Print command help, „all“ for all of them
–enghelp=engine Print ioengine help, or list available ioengines
–enghelp=engine,cmd Print help for an ioengine cmd
–showcmd Turn a job file into command line options
–eta=when When ETA estimate should be printed
May be „always“, „never“ or „auto“
–eta-newline=time Force a new line for every ‚time‘ period passed
–status-interval=t Force full status dump every ‚t‘ period passed
–readonly Turn on safety read-only checks, preventing writes
–section=name Only run specified section in job file
–alloc-size=kb Set smalloc pool to this size in kb (def 1024)
–warnings-fatal Fio parser warnings are fatal
–max-jobs=nr Maximum number of threads/processes to support
–server=args Start a backend fio server
–daemonize=pidfile Background fio server, write pid to file
–client=hostname Talk to remote backend fio server at hostname
–idle-prof=option Report cpu idleness on a system or percpu basis
(option=system,percpu) or run unit work
calibration only (option=calibrate)
5. Run fio testing benchmark.
Note: It is recommended to set the following parameters for high throughput.
- –numjobs=16 <number of cores>: Check the number of cores in the host using lscpu
# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 62
Model name: Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
Stepping: 4
CPU MHz: 1200.561
BogoMIPS: 4004.05
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 20480K
NUMA node0 CPU(s): 0-15
- –bs=64k <block size>: Block size. Use higher block size for better throughput.
- –filename=/dev/nvme0n1<device location>: This is the block device used. In this case, it is an NVMe device. In other cases, it could be any other local or remote device.
For other parameters, refer to the fio man page.
Here is an example for using ConnectX-4 Lx adapters:
# fio –bs=64k –numjobs=16 –iodepth=4 –loops=1 –direct=1 –invalidate=1 –fsync_on_close=1 –randrepeat=1 –norandommap –time_based –runtime=60 –filename=/dev/nvme0n1 –name=read-phase –rw=randread
read-phase: (g=0): rw=randread, bs=64K-64K/64K-64K/64K-64K, ioengine=sync, iodepth=4
…
fio-2.1.10
Starting 16 processes
Jobs: 16 (f=16): [rrrrrrrrrrrrrrrr] [18.0% done] [2722MB/0KB/0KB /s] [43.6K/0/0 iops] [eta 00m:50s]
