关于磁盘IO的那些事

一、背景

需要把生产上一份clickhouse的备份数据到测试环境上，心想也就59G 的大小，应该很快吧，殊不知压缩就搞了3个多小时，于是好奇看下磁盘IO的一些参数，分享给大家学习参考

在分析存储性能之前，需要先了解存储性能定义的三个核心指标：IOPS、Throughput、await

指标	说明
IOPS	每秒进行的IO操作次数
Throughput[吞吐量]	每秒的I/O请求大小
await[响应时间]	指I/O请求从发出到收到响应的间隔时间

二、使用fio工具进行磁盘性能测试

1、FIO是一款开源的 I/O 压力测试工具，主要是用来测试磁盘的IO性能。它可以支持多种不同的I/O引擎

fio官方文档：https://fio.readthedocs.io/en/latest/index.html

2、安装

[root@sre ~]# yum provides fio
已加载插件：fastestmirror
Loading mirror speeds from cached hostfile
fio-3.7-2.el7.x86_64 : Multithreaded IO generation tool
源    ：base

[root@sre ~]# yum install fio-3.7-2.el7.x86_64 -y

3、参数

name 要测试的文件

filename=/dev/emcpowerb　支持文件系统或者裸设备，-filename=/dev/sda2或-filename=/dev/sdb

direct=1 测试过程绕过机器自带的buffer，使测试结果更真实

thread 创建线程为POSIX线程

rw=randwread 测试随机读的I/O

rw=randwrite 测试随机写的I/O

rw=randrw I/O类型为测试随机混合写和读的I/O，默认写占30%，读占70%

rw=read 测试顺序读的I/O

rw=write 测试顺序写的I/O

rw=rw 测试顺序混合写和读的I/O

bs=4k 单次io块大小为4k

size 本次的测试文件大小为500M，以每次4k的io进行测试

iodepth=1 队列深度，这是一个可以影响IOPS的参数，增加队列深度，可以测试出磁盘的峰值

numjobs=6 总共开6个线程读

ioengine=psync io引擎使用pync方式，如果要使用libaio引擎，需要yum install libaio-devel包

runtime: 总共运行时间为10秒，如果不写则一直将500M文件分4k每次写完为止

group_reporting 关于显示结果的，汇总每个进程的信息

4、其他参数

lockmem=1g 只使用1g内存进行测试

zero_buffers 用0初始化系统buffer

nrfiles=8 每个进程生成文件的数量

5、磁盘读写常用测试点

Read=100% Sequence=100% rw=read （100%顺序读）
Write=100% Sequence=100% rw=write （100%顺序写）
Read=70% Sequence=100% rw=rw, rwmixread=70, rwmixwrite=30（70%顺序读，30%顺序写）
Read=100% Ramdon=100% rw=randread (100%随机读)
Write=100% Ramdon=100% rw=randwrite （100%随机写）
Read=70% Ramdon=100% rw=randrw, rwmixread=70, rwmixwrite=30 (70%随机读，30%随机写)

6、FIO 工具测试模式：

测试模式	场景
顺序读写	进行大容量文件读写时，读写过程中会遵循先后顺序，数值越高代表读写性能越强，主要针对大容量文件。
随机读写	不遵循文件的先后顺序，读写时可以任意跳到某个文件。性能主要针对是小文件相关读写任务

7、测试样例

使用FIO进行读写操作为高危操作，请务必提前查询和确认好用于FIO测试的卷挂载详情，并在后续测试过程中，严格使用查询到的测试卷设备名，避免出现对系统盘或者其他业务数据卷进行误操作

（1）、100%，100%顺序读，4K

[root@sre ~]# fio -name=sqe_100read_4k -filename=/dev/sda2  -direct=1 -iodepth=1  -thread -rw=read -ioengine=psync -bs=4k -size=1G -numjobs=10  -runtime=120 -group_reporting
sqe_100read_4k: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 10 threads
Jobs: 10 (f=10): [R(10)][100.0%][r=137MiB/s,w=0KiB/s][r=35.1k,w=0 IOPS][eta 00m:00s]
sqe_100read_4k: (groupid=0, jobs=10): err= 0: pid=30672: Fri Feb 24 13:02:53 2023
   read: IOPS=9002, BW=35.2MiB/s (36.9MB/s)(4220MiB/120005msec)
    clat (usec): min=11, max=835604, avg=1109.96, stdev=8600.91
     lat (usec): min=11, max=835604, avg=1110.08, stdev=8600.92
    clat percentiles (usec):
     |  1.00th=[    37],  5.00th=[    64], 10.00th=[    82], 20.00th=[   105],
     | 30.00th=[   120], 40.00th=[   133], 50.00th=[   147], 60.00th=[   161],
     | 70.00th=[   184], 80.00th=[   217], 90.00th=[   734], 95.00th=[  5669],
     | 99.00th=[ 13304], 99.50th=[ 24511], 99.90th=[123208], 99.95th=[168821],
     | 99.99th=[308282]
   bw (  KiB/s): min=    8, max=19456, per=10.02%, avg=3606.87, stdev=4579.11, samples=2392
   iops        : min=    2, max= 4864, avg=901.69, stdev=1144.79, samples=2392
  lat (usec)   : 20=0.01%, 50=2.58%, 100=14.34%, 250=67.72%, 500=3.93%
  lat (usec)   : 750=1.50%, 1000=1.08%
  lat (msec)   : 2=1.40%, 4=1.35%, 10=4.21%, 20=1.27%, 50=0.34%
  lat (msec)   : 100=0.15%, 250=0.10%, 500=0.02%, 750=0.01%, 1000=0.01%
  cpu          : usr=0.21%, sys=0.89%, ctx=1079963, majf=0, minf=21
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, &gt;=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0%
     issued rwts: total=1080317,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=35.2MiB/s (36.9MB/s), 35.2MiB/s-35.2MiB/s (36.9MB/s-36.9MB/s), io=4220MiB (4425MB), run=120005-120005msec

Disk stats (read/write):
  sda: ios=1078390/8445, merge=0/31, ticks=1188172/367350, in_queue=1555389, util=100.00%

（2）、100%，100%顺序写，4K

[root@sre ~]# fio -name=sqe_100write_4k -filename=/dev/sda2  -direct=1 -iodepth=1  -thread -rw=write -ioengine=psync -bs=4k -size=1G -numjobs=10  -runtime=120 -group_reporting
sqe_100write_4k: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 10 threads
Jobs: 10 (f=10): [W(10)][100.0%][r=0KiB/s,w=159MiB/s][r=0,w=40.8k IOPS][eta 00m:00s]
sqe_100write_4k: (groupid=0, jobs=10): err= 0: pid=32135: Fri Feb 24 13:07:20 2023
  write: IOPS=34.5k, BW=135MiB/s (141MB/s)(10.0GiB/75895msec)
    clat (usec): min=51, max=1688.4k, avg=288.56, stdev=6133.39
     lat (usec): min=51, max=1688.4k, avg=288.74, stdev=6133.39
    clat percentiles (usec):
     |  1.00th=[    78],  5.00th=[    90], 10.00th=[    98], 20.00th=[   110],
     | 30.00th=[   119], 40.00th=[   126], 50.00th=[   135], 60.00th=[   143],
     | 70.00th=[   153], 80.00th=[   167], 90.00th=[   204], 95.00th=[   253],
     | 99.00th=[  3261], 99.50th=[  5145], 99.90th=[ 12518], 99.95th=[ 34866],
     | 99.99th=[202376]
   bw (  KiB/s): min=    8, max=39896, per=10.48%, avg=14474.40, stdev=8454.32, samples=1440
   iops        : min=    2, max= 9974, avg=3618.58, stdev=2113.59, samples=1440
  lat (usec)   : 100=11.56%, 250=83.19%, 500=3.79%, 750=0.14%, 1000=0.05%
  lat (msec)   : 2=0.05%, 4=0.32%, 10=0.78%, 20=0.06%, 50=0.03%
  lat (msec)   : 100=0.01%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
  cpu          : usr=0.79%, sys=4.06%, ctx=2621452, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, &gt;=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0%
     issued rwts: total=0,2621440,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=135MiB/s (141MB/s), 135MiB/s-135MiB/s (141MB/s-141MB/s), io=10.0GiB (10.7GB), run=75895-75895msec

Disk stats (read/write):
  sda: ios=109/2623562, merge=0/28, ticks=4282/910098, in_queue=914111, util=100.00%

（3）、100%，70%顺序读，30%顺序写 4K

[root@sre ~]# fio -name=sqe_rw_70read_4k -filename=/dev/sda2  -direct=1 -iodepth=1  -thread -rw=rw -rwmixread=70 -ioengine=psync -bs=4k -size=1G -numjobs=10  -runtime=120 -group_reporting
sqe_rw_70read_4k: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 10 threads
Jobs: 10 (f=10): [M(10)][100.0%][r=24.1MiB/s,w=9.99MiB/s][r=6172,w=2556 IOPS][eta 00m:00s] 
sqe_rw_70read_4k: (groupid=0, jobs=10): err= 0: pid=3512: Fri Feb 24 13:24:03 2023
   read: IOPS=6946, BW=27.1MiB/s (28.5MB/s)(3258MiB/120049msec)
    clat (usec): min=44, max=1405.9k, avg=1219.85, stdev=8850.12
     lat (usec): min=44, max=1405.9k, avg=1219.98, stdev=8850.13
    clat percentiles (usec):
     |  1.00th=[    62],  5.00th=[    78], 10.00th=[    89], 20.00th=[   104],
     | 30.00th=[   117], 40.00th=[   128], 50.00th=[   141], 60.00th=[   155],
     | 70.00th=[   178], 80.00th=[   229], 90.00th=[  1401], 95.00th=[  6980],
     | 99.00th=[ 14877], 99.50th=[ 25822], 99.90th=[ 98042], 99.95th=[133694],
     | 99.99th=[265290]
   bw (  KiB/s): min=   56, max=14016, per=10.12%, avg=2812.14, stdev=2999.01, samples=2371
   iops        : min=   14, max= 3504, avg=702.97, stdev=749.75, samples=2371
  write: IOPS=2980, BW=11.6MiB/s (12.2MB/s)(1398MiB/120049msec)
    clat (usec): min=52, max=491303, avg=507.44, stdev=5171.58
     lat (usec): min=52, max=491304, avg=507.65, stdev=5171.58
    clat percentiles (usec):
     |  1.00th=[    72],  5.00th=[    86], 10.00th=[   101], 20.00th=[   126],
     | 30.00th=[   145], 40.00th=[   161], 50.00th=[   176], 60.00th=[   192],
     | 70.00th=[   212], 80.00th=[   237], 90.00th=[   293], 95.00th=[   490],
     | 99.00th=[  4817], 99.50th=[ 11994], 99.90th=[ 83362], 99.95th=[117965],
     | 99.99th=[208667]
   bw (  KiB/s): min=   15, max= 6128, per=10.12%, avg=1206.56, stdev=1287.84, samples=2371
   iops        : min=    3, max= 1532, avg=301.57, stdev=321.97, samples=2371
  lat (usec)   : 50=0.01%, 100=14.76%, 250=67.61%, 500=6.31%, 750=2.02%
  lat (usec)   : 1000=1.10%
  lat (msec)   : 2=1.28%, 4=1.17%, 10=3.97%, 20=1.19%, 50=0.35%
  lat (msec)   : 100=0.15%, 250=0.08%, 500=0.01%, 750=0.01%, 1000=0.01%
  cpu          : usr=0.24%, sys=0.99%, ctx=1191799, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, &gt;=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0%
     issued rwts: total=833936,357813,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=27.1MiB/s (28.5MB/s), 27.1MiB/s-27.1MiB/s (28.5MB/s-28.5MB/s), io=3258MiB (3416MB), run=120049-120049msec
  WRITE: bw=11.6MiB/s (12.2MB/s), 11.6MiB/s-11.6MiB/s (12.2MB/s-12.2MB/s), io=1398MiB (1466MB), run=120049-120049msec

Disk stats (read/write):
  sda: ios=831488/361338, merge=0/71, ticks=998352/328430, in_queue=1326637, util=100.00%

（4）、100%，100%随机读，4K

[root@sre ~]# fio -name=rand_100read_4k -filename=/dev/sda2  -direct=1 -iodepth=1  -thread -rw=randread -ioengine=psync -bs=4k -size=1G -numjobs=10  -runtime=120 -group_reporting
rand_100read_4k: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 10 threads
Jobs: 10 (f=10): [r(10)][100.0%][r=521KiB/s,w=0KiB/s][r=130,w=0 IOPS][eta 00m:00s]
rand_100read_4k: (groupid=0, jobs=10): err= 0: pid=10961: Fri Feb 24 14:16:54 2023
   read: IOPS=420, BW=1683KiB/s (1724kB/s)(197MiB/120022msec)
    clat (usec): min=73, max=976990, avg=23750.11, stdev=41850.90
     lat (usec): min=73, max=976991, avg=23750.48, stdev=41850.92
    clat percentiles (usec):
     |  1.00th=[   412],  5.00th=[  2180], 10.00th=[  4752], 20.00th=[  7177],
     | 30.00th=[  9110], 40.00th=[ 10945], 50.00th=[ 12911], 60.00th=[ 15533],
     | 70.00th=[ 19268], 80.00th=[ 25822], 90.00th=[ 43254], 95.00th=[ 82314],
     | 99.00th=[217056], 99.50th=[291505], 99.90th=[455082], 99.95th=[541066],
     | 99.99th=[784335]
   bw (  KiB/s): min=    7, max=  400, per=10.00%, avg=168.34, stdev=89.05, samples=2393
   iops        : min=    1, max=  100, avg=41.85, stdev=22.29, samples=2393
  lat (usec)   : 100=0.01%, 250=0.08%, 500=1.99%, 750=1.12%, 1000=0.75%
  lat (msec)   : 2=0.91%, 4=3.10%, 10=27.07%, 20=36.74%, 50=19.71%
  lat (msec)   : 100=4.53%, 250=3.24%, 500=0.69%, 750=0.05%, 1000=0.02%
  cpu          : usr=0.03%, sys=0.13%, ctx=50535, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, &gt;=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0%
     issued rwts: total=50512,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=1683KiB/s (1724kB/s), 1683KiB/s-1683KiB/s (1724kB/s-1724kB/s), io=197MiB (207MB), run=120022-120022msec

Disk stats (read/write):
  sda: ios=51029/15975, merge=10/120, ticks=1228499/486616, in_queue=1715958, util=100.00%

（5）、100%，100%随机写，4K

[root@sre ~]# fio -name=rand_100write_4k -filename=/dev/sda2  -direct=1 -iodepth=1  -thread -rw=randwrite -ioengine=psync -bs=4k -size=1G -numjobs=10  -runtime=120 -group_reporting
rand_100write_4k: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 10 threads
Jobs: 10 (f=10): [w(10)][100.0%][r=0KiB/s,w=10.7MiB/s][r=0,w=2747 IOPS][eta 00m:00s]
rand_100write_4k: (groupid=0, jobs=10): err= 0: pid=11803: Fri Feb 24 13:29:35 2023
  write: IOPS=1188, BW=4755KiB/s (4869kB/s)(558MiB/120134msec)
    clat (usec): min=54, max=1692.6k, avg=8404.99, stdev=42966.47
     lat (usec): min=54, max=1692.6k, avg=8405.21, stdev=42966.47
    clat percentiles (usec):
     |  1.00th=[    103],  5.00th=[    133], 10.00th=[    157],
     | 20.00th=[    200], 30.00th=[    255], 40.00th=[    322],
     | 50.00th=[    404], 60.00th=[    529], 70.00th=[    734],
     | 80.00th=[   1352], 90.00th=[   8455], 95.00th=[  41157],
     | 99.00th=[ 164627], 99.50th=[ 246416], 99.90th=[ 517997],
     | 99.95th=[ 750781], 99.99th=[1417675]
   bw (  KiB/s): min=    7, max= 4088, per=10.46%, avg=497.31, stdev=463.38, samples=2295
   iops        : min=    1, max= 1022, avg=124.17, stdev=115.89, samples=2295
  lat (usec)   : 100=0.75%, 250=28.39%, 500=29.08%, 750=12.21%, 1000=5.44%
  lat (msec)   : 2=8.18%, 4=3.49%, 10=3.13%, 20=2.38%, 50=2.45%
  lat (msec)   : 100=2.01%, 250=2.01%, 500=0.38%, 750=0.06%, 1000=0.03%
  cpu          : usr=0.04%, sys=0.15%, ctx=142837, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, &gt;=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0%
     issued rwts: total=0,142815,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=4755KiB/s (4869kB/s), 4755KiB/s-4755KiB/s (4869kB/s-4869kB/s), io=558MiB (585MB), run=120134-120134msec

Disk stats (read/write):
  sda: ios=40/148476, merge=17/74, ticks=18004/1646246, in_queue=1668467, util=100.00%

（6）、100%，70%随机读，30%随机写 4K

[root@sre ~]# fio -name=rand_rw_70read_4k -filename=/dev/sda2  -direct=1 -iodepth=1  -thread -rw=rw -rwmixread=70 -ioengine=psync -bs=4k -size=1G -numjobs=10  -runtime=120 -group_reporting
rand_rw_70read_4k: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 10 threads
Jobs: 10 (f=10): [M(10)][100.0%][r=10.1MiB/s,w=4670KiB/s][r=2591,w=1167 IOPS][eta 00m:00s] 
rand_rw_70read_4k: (groupid=0, jobs=10): err= 0: pid=26920: Fri Feb 24 13:34:23 2023
   read: IOPS=6645, BW=25.0MiB/s (27.2MB/s)(3115MiB/120003msec)
    clat (usec): min=38, max=610563, avg=1279.77, stdev=8021.32
     lat (usec): min=38, max=610563, avg=1279.90, stdev=8021.33
    clat percentiles (usec):
     |  1.00th=[    53],  5.00th=[    63], 10.00th=[    73], 20.00th=[    88],
     | 30.00th=[   100], 40.00th=[   113], 50.00th=[   127], 60.00th=[   145],
     | 70.00th=[   176], 80.00th=[   330], 90.00th=[  1418], 95.00th=[  6915],
     | 99.00th=[ 15795], 99.50th=[ 29754], 99.90th=[119014], 99.95th=[160433],
     | 99.99th=[325059]
   bw (  KiB/s): min=    8, max=16128, per=10.00%, avg=2657.88, stdev=3002.47, samples=2389
   iops        : min=    2, max= 4032, avg=664.31, stdev=750.64, samples=2389
  write: IOPS=2849, BW=11.1MiB/s (11.7MB/s)(1336MiB/120003msec)
    clat (usec): min=46, max=419565, avg=520.42, stdev=4599.74
     lat (usec): min=46, max=419565, avg=520.62, stdev=4599.75
    clat percentiles (usec):
     |  1.00th=[    63],  5.00th=[    74], 10.00th=[    84], 20.00th=[   105],
     | 30.00th=[   125], 40.00th=[   143], 50.00th=[   161], 60.00th=[   182],
     | 70.00th=[   210], 80.00th=[   262], 90.00th=[   562], 95.00th=[   799],
     | 99.00th=[  6259], 99.50th=[ 11076], 99.90th=[ 56361], 99.95th=[ 90702],
     | 99.99th=[229639]
   bw (  KiB/s): min=    7, max= 6912, per=10.10%, avg=1151.11, stdev=1288.82, samples=2365
   iops        : min=    1, max= 1728, avg=287.61, stdev=322.25, samples=2365
  lat (usec)   : 50=0.22%, 100=26.02%, 250=51.97%, 500=6.21%, 750=5.09%
  lat (usec)   : 1000=1.68%
  lat (msec)   : 2=1.70%, 4=1.39%, 10=3.74%, 20=1.39%, 50=0.35%
  lat (msec)   : 100=0.15%, 250=0.09%, 500=0.01%, 750=0.01%
  cpu          : usr=0.19%, sys=0.74%, ctx=1139413, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, &gt;=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0%
     issued rwts: total=797444,341945,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=25.0MiB/s (27.2MB/s), 25.0MiB/s-25.0MiB/s (27.2MB/s-27.2MB/s), io=3115MiB (3266MB), run=120003-120003msec
  WRITE: bw=11.1MiB/s (11.7MB/s), 11.1MiB/s-11.1MiB/s (11.7MB/s-11.7MB/s), io=1336MiB (1401MB), run=120003-120003msec

Disk stats (read/write):
  sda: ios=796912/350573, merge=0/72, ticks=914709/402509, in_queue=1317165, util=99.50%

测试结果：

1、磁盘的每秒读写次数iops，这个是随机读写考察的重点;

2、磁盘的吞吐量bw(MB/s)，这个是顺序读写考察的重点;

3、await响应时间[usec代表微秒]，avg平均响应时间）值越小,性能越好

测试模式(4K)	iops	bw(MB/s)	await(ms)
顺序读	9002	35.2	1.109
顺序写	34500	135	0.288
70%顺序读，30%顺序写	6946	27.1	1.219
随机读	420	1.6	23.750
随机写	1188	4.7	8.404
70%随机读，30%随机写	6645	25	1.279

由于4K随机读写速度有两种单位IOPS和MB/s，为方便我们比较，可以通过用下面公式来取得一个近似值。

总结以上论证就是说：存储的性能是由三个核心指标：iops、bw(Throughput)、await来定义存储是不是有性能瓶颈，通过FIO工具可以很直观的展现

三、观测磁盘I/O

1、iostat是常见的磁盘I/O性能观测工具，运维通过命令实时查看磁盘负荷情况

2、安装

yum install sysstat

3、参数

-d: 显示该设备的状态的参数

-x：是一个比较常用的选项，该选项将用于显示和io相关的扩展数据

-m: 以M为单位显示所有信息

-d: sda/sdb 显示指定硬盘信息

-t: 报告每秒向终端读取和写入的字符数

数字2 ：表示几秒刷新一次

[root@sre ~]# iostat -mtxd 2
Linux 3.10.0-1127.el7.x86_64 (sre)  2023年02月24日     _x86_64_    (8 CPU)

2023年02月24日 17时19分51秒
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.01     0.32    1.63   69.04     0.13     1.27    40.54     0.17    2.37  161.48    3.30   2.03  14.34
scd0              0.00     0.00    0.00    0.00     0.00     0.00   114.22     0.00    0.11    0.11    0.00   0.11   0.00
dm-0              0.00     0.00    0.01    0.63     0.00     0.02    64.84     0.02   30.87   79.49   30.18   7.08   0.45
dm-1              0.00     0.00    0.01    0.02     0.00     0.00     8.01     0.00  138.56   43.93  193.34   4.93   0.01
dm-2              0.00     0.00    1.37   68.44     0.12     1.25    40.39     0.15    2.18  191.04    3.13   2.03  14.19

2023年02月24日 17时19分53秒
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00   56.50     0.00     0.56    20.13     0.10    1.77    0.00    1.77   0.68   3.85
scd0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-0              0.00     0.00    0.00   35.50     0.00     0.25    14.58     0.02    0.54    0.00    0.54   0.52   1.85
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00   21.00     0.00     0.30    29.52     0.08    3.83    0.00    3.83   1.10   2.30

还是重点关注3个性能核心指标：iops、throughput、await与多个任务指标

网上找了一张图