关于磁盘IO的那些事

关于磁盘IO的那些事

一、背景

需要把生产上一份clickhouse的备份数据到测试环境上,心想也就59G 的大小,应该很快吧,殊不知压缩就搞了3个多小时,于是好奇看下磁盘IO的一些参数,分享给大家学习参考

在分析存储性能之前,需要先了解存储性能定义的三个核心指标:IOPS、Throughput、await

指标 说明
IOPS 每秒进行的IO操作次数
Throughput[吞吐量] 每秒的I/O请求大小
await[响应时间] 指I/O请求从发出到收到响应的间隔时间

二、使用fio工具进行磁盘性能测试

1、FIO是一款开源的 I/O 压力测试工具,主要是用来测试磁盘的IO性能。它可以支持多种不同的I/O引擎

fio官方文档:https://fio.readthedocs.io/en/latest/index.html

2、安装

[root@sre ~]# yum provides fio
已加载插件:fastestmirror
Loading mirror speeds from cached hostfile
fio-3.7-2.el7.x86_64 : Multithreaded IO generation tool
源    :base

[root@sre ~]# yum install fio-3.7-2.el7.x86_64 -y

3、参数

  • name 要测试的文件
  • filename=/dev/emcpowerb 支持文件系统或者裸设备,-filename=/dev/sda2或-filename=/dev/sdb
  • direct=1 测试过程绕过机器自带的buffer,使测试结果更真实
  • thread 创建线程为POSIX线程
  • rw=randwread 测试随机读的I/O
  • rw=randwrite 测试随机写的I/O
  • rw=randrw I/O类型为测试随机混合写和读的I/O,默认写占30%,读占70%
  • rw=read 测试顺序读的I/O
  • rw=write 测试顺序写的I/O
  • rw=rw 测试顺序混合写和读的I/O
  • bs=4k 单次io块大小为4k
  • size 本次的测试文件大小为500M,以每次4k的io进行测试
  • iodepth=1 队列深度,这是一个可以影响IOPS的参数,增加队列深度,可以测试出磁盘的峰值
  • numjobs=6 总共开6个线程读
  • ioengine=psync io引擎使用pync方式,如果要使用libaio引擎,需要yum install libaio-devel包
  • runtime: 总共运行时间为10秒,如果不写则一直将500M文件分4k每次写完为止
  • group_reporting 关于显示结果的,汇总每个进程的信息

4、其他参数

  • lockmem=1g 只使用1g内存进行测试
  • zero_buffers 用0初始化系统buffer
  • nrfiles=8 每个进程生成文件的数量

5、磁盘读写常用测试点

  • Read=100% Sequence=100% rw=read (100%顺序读)
  • Write=100% Sequence=100% rw=write (100%顺序写)
  • Read=70% Sequence=100% rw=rw, rwmixread=70, rwmixwrite=30(70%顺序读,30%顺序写)
  • Read=100% Ramdon=100% rw=randread (100%随机读)
  • Write=100% Ramdon=100% rw=randwrite (100%随机写)
  • Read=70% Ramdon=100% rw=randrw, rwmixread=70, rwmixwrite=30 (70%随机读,30%随机写)

6、FIO 工具测试模式:

测试模式 场景
顺序读写 进行大容量文件读写时,读写过程中会遵循先后顺序,数值越高代表读写性能越强,主要针对大容量文件。
随机读写 不遵循文件的先后顺序,读写时可以任意跳到某个文件。性能主要针对是小文件相关读写任务

7、测试样例

使用FIO进行读写操作为高危操作,请务必提前查询和确认好用于FIO测试的卷挂载详情,并在后续测试过程中,严格使用查询到的测试卷设备名,避免出现对系统盘或者其他业务数据卷进行误操作

(1)、100%,100%顺序读 ,4K

[root@sre ~]# fio -name=sqe_100read_4k -filename=/dev/sda2  -direct=1 -iodepth=1  -thread -rw=read -ioengine=psync -bs=4k -size=1G -numjobs=10  -runtime=120 -group_reporting
sqe_100read_4k: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 10 threads
Jobs: 10 (f=10): [R(10)][100.0%][r=137MiB/s,w=0KiB/s][r=35.1k,w=0 IOPS][eta 00m:00s]
sqe_100read_4k: (groupid=0, jobs=10): err= 0: pid=30672: Fri Feb 24 13:02:53 2023
   read: IOPS=9002, BW=35.2MiB/s (36.9MB/s)(4220MiB/120005msec)
    clat (usec): min=11, max=835604, avg=1109.96, stdev=8600.91
     lat (usec): min=11, max=835604, avg=1110.08, stdev=8600.92
    clat percentiles (usec):
     |  1.00th=[    37],  5.00th=[    64], 10.00th=[    82], 20.00th=[   105],
     | 30.00th=[   120], 40.00th=[   133], 50.00th=[   147], 60.00th=[   161],
     | 70.00th=[   184], 80.00th=[   217], 90.00th=[   734], 95.00th=[  5669],
     | 99.00th=[ 13304], 99.50th=[ 24511], 99.90th=[123208], 99.95th=[168821],
     | 99.99th=[308282]
   bw (  KiB/s): min=    8, max=19456, per=10.02%, avg=3606.87, stdev=4579.11, samples=2392
   iops        : min=    2, max= 4864, avg=901.69, stdev=1144.79, samples=2392
  lat (usec)   : 20=0.01%, 50=2.58%, 100=14.34%, 250=67.72%, 500=3.93%
  lat (usec)   : 750=1.50%, 1000=1.08%
  lat (msec)   : 2=1.40%, 4=1.35%, 10=4.21%, 20=1.27%, 50=0.34%
  lat (msec)   : 100=0.15%, 250=0.10%, 500=0.02%, 750=0.01%, 1000=0.01%
  cpu          : usr=0.21%, sys=0.89%, ctx=1079963, majf=0, minf=21
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1080317,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=35.2MiB/s (36.9MB/s), 35.2MiB/s-35.2MiB/s (36.9MB/s-36.9MB/s), io=4220MiB (4425MB), run=120005-120005msec

Disk stats (read/write):
  sda: ios=1078390/8445, merge=0/31, ticks=1188172/367350, in_queue=1555389, util=100.00%

(2)、100%,100%顺序写 ,4K

[root@sre ~]# fio -name=sqe_100write_4k -filename=/dev/sda2  -direct=1 -iodepth=1  -thread -rw=write -ioengine=psync -bs=4k -size=1G -numjobs=10  -runtime=120 -group_reporting
sqe_100write_4k: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 10 threads
Jobs: 10 (f=10): [W(10)][100.0%][r=0KiB/s,w=159MiB/s][r=0,w=40.8k IOPS][eta 00m:00s]
sqe_100write_4k: (groupid=0, jobs=10): err= 0: pid=32135: Fri Feb 24 13:07:20 2023
  write: IOPS=34.5k, BW=135MiB/s (141MB/s)(10.0GiB/75895msec)
    clat (usec): min=51, max=1688.4k, avg=288.56, stdev=6133.39
     lat (usec): min=51, max=1688.4k, avg=288.74, stdev=6133.39
    clat percentiles (usec):
     |  1.00th=[    78],  5.00th=[    90], 10.00th=[    98], 20.00th=[   110],
     | 30.00th=[   119], 40.00th=[   126], 50.00th=[   135], 60.00th=[   143],
     | 70.00th=[   153], 80.00th=[   167], 90.00th=[   204], 95.00th=[   253],
     | 99.00th=[  3261], 99.50th=[  5145], 99.90th=[ 12518], 99.95th=[ 34866],
     | 99.99th=[202376]
   bw (  KiB/s): min=    8, max=39896, per=10.48%, avg=14474.40, stdev=8454.32, samples=1440
   iops        : min=    2, max= 9974, avg=3618.58, stdev=2113.59, samples=1440
  lat (usec)   : 100=11.56%, 250=83.19%, 500=3.79%, 750=0.14%, 1000=0.05%
  lat (msec)   : 2=0.05%, 4=0.32%, 10=0.78%, 20=0.06%, 50=0.03%
  lat (msec)   : 100=0.01%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
  cpu          : usr=0.79%, sys=4.06%, ctx=2621452, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2621440,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=135MiB/s (141MB/s), 135MiB/s-135MiB/s (141MB/s-141MB/s), io=10.0GiB (10.7GB), run=75895-75895msec

Disk stats (read/write):
  sda: ios=109/2623562, merge=0/28, ticks=4282/910098, in_queue=914111, util=100.00%

(3)、100%,70%顺序读,30%顺序写 4K

[root@sre ~]# fio -name=sqe_rw_70read_4k -filename=/dev/sda2  -direct=1 -iodepth=1  -thread -rw=rw -rwmixread=70 -ioengine=psync -bs=4k -size=1G -numjobs=10  -runtime=120 -group_reporting
sqe_rw_70read_4k: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 10 threads
Jobs: 10 (f=10): [M(10)][100.0%][r=24.1MiB/s,w=9.99MiB/s][r=6172,w=2556 IOPS][eta 00m:00s] 
sqe_rw_70read_4k: (groupid=0, jobs=10): err= 0: pid=3512: Fri Feb 24 13:24:03 2023
   read: IOPS=6946, BW=27.1MiB/s (28.5MB/s)(3258MiB/120049msec)
    clat (usec): min=44, max=1405.9k, avg=1219.85, stdev=8850.12
     lat (usec): min=44, max=1405.9k, avg=1219.98, stdev=8850.13
    clat percentiles (usec):
     |  1.00th=[    62],  5.00th=[    78], 10.00th=[    89], 20.00th=[   104],
     | 30.00th=[   117], 40.00th=[   128], 50.00th=[   141], 60.00th=[   155],
     | 70.00th=[   178], 80.00th=[   229], 90.00th=[  1401], 95.00th=[  6980],
     | 99.00th=[ 14877], 99.50th=[ 25822], 99.90th=[ 98042], 99.95th=[133694],
     | 99.99th=[265290]
   bw (  KiB/s): min=   56, max=14016, per=10.12%, avg=2812.14, stdev=2999.01, samples=2371
   iops        : min=   14, max= 3504, avg=702.97, stdev=749.75, samples=2371
  write: IOPS=2980, BW=11.6MiB/s (12.2MB/s)(1398MiB/120049msec)
    clat (usec): min=52, max=491303, avg=507.44, stdev=5171.58
     lat (usec): min=52, max=491304, avg=507.65, stdev=5171.58
    clat percentiles (usec):
     |  1.00th=[    72],  5.00th=[    86], 10.00th=[   101], 20.00th=[   126],
     | 30.00th=[   145], 40.00th=[   161], 50.00th=[   176], 60.00th=[   192],
     | 70.00th=[   212], 80.00th=[   237], 90.00th=[   293], 95.00th=[   490],
     | 99.00th=[  4817], 99.50th=[ 11994], 99.90th=[ 83362], 99.95th=[117965],
     | 99.99th=[208667]
   bw (  KiB/s): min=   15, max= 6128, per=10.12%, avg=1206.56, stdev=1287.84, samples=2371
   iops        : min=    3, max= 1532, avg=301.57, stdev=321.97, samples=2371
  lat (usec)   : 50=0.01%, 100=14.76%, 250=67.61%, 500=6.31%, 750=2.02%
  lat (usec)   : 1000=1.10%
  lat (msec)   : 2=1.28%, 4=1.17%, 10=3.97%, 20=1.19%, 50=0.35%
  lat (msec)   : 100=0.15%, 250=0.08%, 500=0.01%, 750=0.01%, 1000=0.01%
  cpu          : usr=0.24%, sys=0.99%, ctx=1191799, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=833936,357813,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=27.1MiB/s (28.5MB/s), 27.1MiB/s-27.1MiB/s (28.5MB/s-28.5MB/s), io=3258MiB (3416MB), run=120049-120049msec
  WRITE: bw=11.6MiB/s (12.2MB/s), 11.6MiB/s-11.6MiB/s (12.2MB/s-12.2MB/s), io=1398MiB (1466MB), run=120049-120049msec

Disk stats (read/write):
  sda: ios=831488/361338, merge=0/71, ticks=998352/328430, in_queue=1326637, util=100.00%

(4)、100%,100%随机读 ,4K

[root@sre ~]# fio -name=rand_100read_4k -filename=/dev/sda2  -direct=1 -iodepth=1  -thread -rw=randread -ioengine=psync -bs=4k -size=1G -numjobs=10  -runtime=120 -group_reporting
rand_100read_4k: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 10 threads
Jobs: 10 (f=10): [r(10)][100.0%][r=521KiB/s,w=0KiB/s][r=130,w=0 IOPS][eta 00m:00s]
rand_100read_4k: (groupid=0, jobs=10): err= 0: pid=10961: Fri Feb 24 14:16:54 2023
   read: IOPS=420, BW=1683KiB/s (1724kB/s)(197MiB/120022msec)
    clat (usec): min=73, max=976990, avg=23750.11, stdev=41850.90
     lat (usec): min=73, max=976991, avg=23750.48, stdev=41850.92
    clat percentiles (usec):
     |  1.00th=[   412],  5.00th=[  2180], 10.00th=[  4752], 20.00th=[  7177],
     | 30.00th=[  9110], 40.00th=[ 10945], 50.00th=[ 12911], 60.00th=[ 15533],
     | 70.00th=[ 19268], 80.00th=[ 25822], 90.00th=[ 43254], 95.00th=[ 82314],
     | 99.00th=[217056], 99.50th=[291505], 99.90th=[455082], 99.95th=[541066],
     | 99.99th=[784335]
   bw (  KiB/s): min=    7, max=  400, per=10.00%, avg=168.34, stdev=89.05, samples=2393
   iops        : min=    1, max=  100, avg=41.85, stdev=22.29, samples=2393
  lat (usec)   : 100=0.01%, 250=0.08%, 500=1.99%, 750=1.12%, 1000=0.75%
  lat (msec)   : 2=0.91%, 4=3.10%, 10=27.07%, 20=36.74%, 50=19.71%
  lat (msec)   : 100=4.53%, 250=3.24%, 500=0.69%, 750=0.05%, 1000=0.02%
  cpu          : usr=0.03%, sys=0.13%, ctx=50535, majf=0, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=50512,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=1683KiB/s (1724kB/s), 1683KiB/s-1683KiB/s (1724kB/s-1724kB/s), io=197MiB (207MB), run=120022-120022msec

Disk stats (read/write):
  sda: ios=51029/15975, merge=10/120, ticks=1228499/486616, in_queue=1715958, util=100.00%

(5)、100%,100%随机写 ,4K

[root@sre ~]# fio -name=rand_100write_4k -filename=/dev/sda2  -direct=1 -iodepth=1  -thread -rw=randwrite -ioengine=psync -bs=4k -size=1G -numjobs=10  -runtime=120 -group_reporting
rand_100write_4k: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 10 threads
Jobs: 10 (f=10): [w(10)][100.0%][r=0KiB/s,w=10.7MiB/s][r=0,w=2747 IOPS][eta 00m:00s]
rand_100write_4k: (groupid=0, jobs=10): err= 0: pid=11803: Fri Feb 24 13:29:35 2023
  write: IOPS=1188, BW=4755KiB/s (4869kB/s)(558MiB/120134msec)
    clat (usec): min=54, max=1692.6k, avg=8404.99, stdev=42966.47
     lat (usec): min=54, max=1692.6k, avg=8405.21, stdev=42966.47
    clat percentiles (usec):
     |  1.00th=[    103],  5.00th=[    133], 10.00th=[    157],
     | 20.00th=[    200], 30.00th=[    255], 40.00th=[    322],
     | 50.00th=[    404], 60.00th=[    529], 70.00th=[    734],
     | 80.00th=[   1352], 90.00th=[   8455], 95.00th=[  41157],
     | 99.00th=[ 164627], 99.50th=[ 246416], 99.90th=[ 517997],
     | 99.95th=[ 750781], 99.99th=[1417675]
   bw (  KiB/s): min=    7, max= 4088, per=10.46%, avg=497.31, stdev=463.38, samples=2295
   iops        : min=    1, max= 1022, avg=124.17, stdev=115.89, samples=2295
  lat (usec)   : 100=0.75%, 250=28.39%, 500=29.08%, 750=12.21%, 1000=5.44%
  lat (msec)   : 2=8.18%, 4=3.49%, 10=3.13%, 20=2.38%, 50=2.45%
  lat (msec)   : 100=2.01%, 250=2.01%, 500=0.38%, 750=0.06%, 1000=0.03%
  cpu          : usr=0.04%, sys=0.15%, ctx=142837, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,142815,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=4755KiB/s (4869kB/s), 4755KiB/s-4755KiB/s (4869kB/s-4869kB/s), io=558MiB (585MB), run=120134-120134msec

Disk stats (read/write):
  sda: ios=40/148476, merge=17/74, ticks=18004/1646246, in_queue=1668467, util=100.00%

(6)、100%,70%随机读,30%随机写 4K

[root@sre ~]# fio -name=rand_rw_70read_4k -filename=/dev/sda2  -direct=1 -iodepth=1  -thread -rw=rw -rwmixread=70 -ioengine=psync -bs=4k -size=1G -numjobs=10  -runtime=120 -group_reporting
rand_rw_70read_4k: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 10 threads
Jobs: 10 (f=10): [M(10)][100.0%][r=10.1MiB/s,w=4670KiB/s][r=2591,w=1167 IOPS][eta 00m:00s] 
rand_rw_70read_4k: (groupid=0, jobs=10): err= 0: pid=26920: Fri Feb 24 13:34:23 2023
   read: IOPS=6645, BW=25.0MiB/s (27.2MB/s)(3115MiB/120003msec)
    clat (usec): min=38, max=610563, avg=1279.77, stdev=8021.32
     lat (usec): min=38, max=610563, avg=1279.90, stdev=8021.33
    clat percentiles (usec):
     |  1.00th=[    53],  5.00th=[    63], 10.00th=[    73], 20.00th=[    88],
     | 30.00th=[   100], 40.00th=[   113], 50.00th=[   127], 60.00th=[   145],
     | 70.00th=[   176], 80.00th=[   330], 90.00th=[  1418], 95.00th=[  6915],
     | 99.00th=[ 15795], 99.50th=[ 29754], 99.90th=[119014], 99.95th=[160433],
     | 99.99th=[325059]
   bw (  KiB/s): min=    8, max=16128, per=10.00%, avg=2657.88, stdev=3002.47, samples=2389
   iops        : min=    2, max= 4032, avg=664.31, stdev=750.64, samples=2389
  write: IOPS=2849, BW=11.1MiB/s (11.7MB/s)(1336MiB/120003msec)
    clat (usec): min=46, max=419565, avg=520.42, stdev=4599.74
     lat (usec): min=46, max=419565, avg=520.62, stdev=4599.75
    clat percentiles (usec):
     |  1.00th=[    63],  5.00th=[    74], 10.00th=[    84], 20.00th=[   105],
     | 30.00th=[   125], 40.00th=[   143], 50.00th=[   161], 60.00th=[   182],
     | 70.00th=[   210], 80.00th=[   262], 90.00th=[   562], 95.00th=[   799],
     | 99.00th=[  6259], 99.50th=[ 11076], 99.90th=[ 56361], 99.95th=[ 90702],
     | 99.99th=[229639]
   bw (  KiB/s): min=    7, max= 6912, per=10.10%, avg=1151.11, stdev=1288.82, samples=2365
   iops        : min=    1, max= 1728, avg=287.61, stdev=322.25, samples=2365
  lat (usec)   : 50=0.22%, 100=26.02%, 250=51.97%, 500=6.21%, 750=5.09%
  lat (usec)   : 1000=1.68%
  lat (msec)   : 2=1.70%, 4=1.39%, 10=3.74%, 20=1.39%, 50=0.35%
  lat (msec)   : 100=0.15%, 250=0.09%, 500=0.01%, 750=0.01%
  cpu          : usr=0.19%, sys=0.74%, ctx=1139413, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=797444,341945,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=25.0MiB/s (27.2MB/s), 25.0MiB/s-25.0MiB/s (27.2MB/s-27.2MB/s), io=3115MiB (3266MB), run=120003-120003msec
  WRITE: bw=11.1MiB/s (11.7MB/s), 11.1MiB/s-11.1MiB/s (11.7MB/s-11.7MB/s), io=1336MiB (1401MB), run=120003-120003msec

Disk stats (read/write):
  sda: ios=796912/350573, merge=0/72, ticks=914709/402509, in_queue=1317165, util=99.50%

测试结果:

1、磁盘的每秒读写次数iops,这个是随机读写考察的重点;

2、磁盘的吞吐量bw(MB/s),这个是顺序读写考察的重点;

3、await响应时间[usec代表微秒],avg平均响应时间)值越小,性能越好

测试模式(4K) iops bw(MB/s) await(ms)
顺序读 9002 35.2 1.109
顺序写 34500 135 0.288
70%顺序读,30%顺序写 6946 27.1 1.219
随机读 420 1.6 23.750
随机写 1188 4.7 8.404
70%随机读,30%随机写 6645 25 1.279

由于4K随机读写速度有两种单位IOPS和MB/s,为方便我们比较,可以通过用下面公式来取得一个近似值。

关于磁盘IO的那些事

总结以上论证就是说:存储的性能是由三个核心指标:iops、bw(Throughput)、await来定义存储是不是有性能瓶颈,通过FIO工具可以很直观的展现

三、观测磁盘I/O

1、iostat是常见的磁盘I/O性能观测工具,运维通过命令实时查看磁盘负荷情况

2、安装

yum install sysstat

3、参数

-d: 显示该设备的状态的参数

-x:是一个比较常用的选项,该选项将用于显示和io相关的扩展数据

-m: 以M为单位显示所有信息

-d: sda/sdb 显示指定硬盘信息

-t: 报告每秒向终端读取和写入的字符数

数字2 : 表示几秒刷新一次

[root@sre ~]# iostat -mtxd 2
Linux 3.10.0-1127.el7.x86_64 (sre)  2023年02月24日     _x86_64_    (8 CPU)

2023年02月24日 17时19分51秒
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.01     0.32    1.63   69.04     0.13     1.27    40.54     0.17    2.37  161.48    3.30   2.03  14.34
scd0              0.00     0.00    0.00    0.00     0.00     0.00   114.22     0.00    0.11    0.11    0.00   0.11   0.00
dm-0              0.00     0.00    0.01    0.63     0.00     0.02    64.84     0.02   30.87   79.49   30.18   7.08   0.45
dm-1              0.00     0.00    0.01    0.02     0.00     0.00     8.01     0.00  138.56   43.93  193.34   4.93   0.01
dm-2              0.00     0.00    1.37   68.44     0.12     1.25    40.39     0.15    2.18  191.04    3.13   2.03  14.19

2023年02月24日 17时19分53秒
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00   56.50     0.00     0.56    20.13     0.10    1.77    0.00    1.77   0.68   3.85
scd0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-0              0.00     0.00    0.00   35.50     0.00     0.25    14.58     0.02    0.54    0.00    0.54   0.52   1.85
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00   21.00     0.00     0.30    29.52     0.08    3.83    0.00    3.83   1.10   2.30

还是重点关注3个性能核心指标:iops、throughput、await与多个任务指标

网上找了一张图

关于磁盘IO的那些事

如何通过以上指标数据判断磁盘有没有瓶颈呢?

(1)、如果 %util 接近 100%,说明产生的I/O请求太多,I/O系统已经满负荷,该磁盘可能存在瓶颈

(2)、如果 svctm的值与await很接近,说明IO任务提交之后,IO立即响应了,表示几乎没有I/O等待,磁盘性能很好,反之则有问题;

(3)、查看aqu-sz队列值,越小越好,反之差值越大,队列越大,说明有问题;

四、参考

文章来源:https://www.cnaaa.net,转载请注明出处:https://www.cnaaa.net/archives/7941

(0)
杰斯的头像杰斯
上一篇 2023年4月11日 下午6:04
下一篇 2023年4月12日 下午4:03

相关推荐

  • 二进制安装MySQL

    Yum/rpm包安装,简单、快速、无法定制化、新手推荐使用 二进制安装,解压缩后直接简单配置即可使用,速度较快,专业DBA常用 源码编译安装,特点是可以定制化安装需求,缺点过程较为复杂 二进制安装 创建mysql用户 因为这个用户只是用来进行管理,为了系统安全,配置无登录权限 下载mysql二进制软件包(本案例为mysql5.7.33版本) 创建tools目…

    2022年6月24日
    40600
  • 在Proxmox VE下开启vGPU – Tesla P4为例

    对于 Proxmox VE 8,需要使用16.0+版本的 vGPU 驱动程序,低版本的驱动程序不支持 Linux 6.x 内核 最近我刚升级了我的 NAS 主机,并且购买了一块 Tesla P4 显卡。我想把 vGPU 分配给虚拟机上的 Jellyfin 容器硬解使用,经过几天的研究,我简单总结了使用方法。关于如何使用硬解,我会在下一篇文章中进行说明。 术语…

    2024年2月19日
    21900
  • lscpu 查看 CPU 信息

    lscpu 显示 CPU 的架构信息 lscpu 从 sysfs 和 proc/cpuinfo 中收集信息。这个命令的输出是规范的可以用来解析,或者给人来阅读。该命令显示的信息包括,CPU 的数量,线程 (thread),核心 (core),Socket 还有 Non-Uniform Me…

    2023年9月25日
    25900
  • MySQL5.7主从同步、读写分离

    MySQL5.7主从同步、读写分离 安装MySQL5.7 下载rpm包 安装 启动mysql查看密码 登录mysql 主从同步 关闭每台服务器的防火墙和selinux 主从服务器时间同步 主服务器使用本地时钟源 修改配置文件 开启ntpd 从服务器同步主服务器时间 安装ntp工具 启动ntp和主服务器进行时间同步 主服务器mysql配置 修改配置文件 重启m…

    2022年7月31日
    54400
  • Key exchange was not finished, connection is closed.解决办法

    利用java连接Linux服务器中碰到错误:Key exchange was not finished, connection is closed.,导致服务器的连接失败,报错如下 原因 是ssh中的key交换算法不兼容导致 解决办法 修改配置文件 添加如下内容 重启服务

    2023年7月25日
    34300

在线咨询: QQ交谈

邮件:712342017@qq.com

工作时间:周一至周五,8:30-17:30,节假日休息

关注微信