Storage-layer I/O performance tests

Introduction

Use the commands in this section to determine if the storage-layer performance meets the NSP cluster deployment requirements. Both tests (storage latency and storage throughput) must be executed and meet or exceed the provided requirements.

Storage latency

In this example, the /test directory is on the same disk where etcd runs.

Enter the following as the root user to run the test:

# fio --rw=write --ioengine=sync --fdatasync=1 --directory=/var/lib/test --size=22m --bs=2300 --name=mytest ↵

The command produces output like the following:

Starting 1 process

mytest: Laying out IO file (1 file / 22MiB)

Jobs: 1 (f=1)

mytest: (groupid=0, jobs=1): err= 0: pid=40944: Mon Jun 15 10:23:23 2020

  write: IOPS=7574, BW=16.6MiB/s (17.4MB/s)(21.0MiB/1324msec)

    clat (usec): min=4, max=261, avg= 9.50, stdev= 4.11

     lat (usec): min=4, max=262, avg= 9.67, stdev= 4.12

    clat percentiles (nsec):

     |  1.00th=[ 5536],  5.00th=[ 5728], 10.00th=[ 5920], 20.00th=[ 6176],

     | 30.00th=[ 7584], 40.00th=[ 8896], 50.00th=[ 9408], 60.00th=[ 9792],

     | 70.00th=[10432], 80.00th=[11584], 90.00th=[12864], 95.00th=[14528],

     | 99.00th=[20352], 99.50th=[23168], 99.90th=[28800], 99.95th=[42752],

     | 99.99th=[60672]

   bw (  KiB/s): min=16868, max=17258, per=100.00%, avg=17063.00, stdev=275.77, samples=2

   iops        : min= 7510, max= 7684, avg=7597.00, stdev=123.04, samples=2

  lat (usec)   : 10=64.21%, 20=34.68%, 50=1.08%, 100=0.02%, 500=0.01%

In the second block of output, which is shown below, the 99.00th percentile must be less than 10 ms. In this test output, the 99.00th percentile is less than 1 ms.

fsync/fdatasync/sync_file_range:

    sync (usec): min=39, max=1174, avg=120.71, stdev=63.89

    sync percentiles (usec):

     |  1.00th=[   42],  5.00th=[   45], 10.00th=[   46], 20.00th=[   48],

     | 30.00th=[   52], 40.00th=[   71], 50.00th=[  153], 60.00th=[  159],

     | 70.00th=[  167], 80.00th=[  178], 90.00th=[  192], 95.00th=[  206],

     | 99.00th=[  229], 99.50th=[  239], 99.90th=[  355], 99.95th=[  416],

     | 99.99th=[  445]

  cpu          : usr=2.95%, sys=29.93%, ctx=15663, majf=0, minf=35

  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     issued rwts: total=0,10029,0,0 short=10029,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=1

Storage throughput

To run this test, first change to the directory where the test is to be performed. The test will create a local file. The output from the command contains read and write IOPS values which must be evaluated against the minimum requirements provided in Table 2-6, Minimum NSP cluster IOPS requirements - local storage

Enter the following as the root user to run the test:

# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read_write.fio --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=50 ↵

The command produces output like the following:

test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64

fio-3.7

Starting 1 process

test: Laying out IO file (1 file / 4096MiB)

Jobs: 1 (f=1): [m(1)][100.0%][r=22.1MiB/s,w=22.2MiB/s][r=5645,w=5674 IOPS][eta 00m:00s]

test: (groupid=0, jobs=1): err= 0: pid=32439: Mon Sep 21 10:25:11 2020

read: IOPS=6301, BW=24.6MiB/s (25.8MB/s)(2049MiB/83252msec)

   bw (  KiB/s): min=13824, max=39088, per=99.57%, avg=25098.60, stdev=5316.27, samples=166

   iops        : min= 3456, max= 9772, avg=6274.49, stdev=1329.11, samples=166

write: IOPS=6293, BW=24.6MiB/s (25.8MB/s)(2047MiB/83252msec)

   bw (  KiB/s): min=13464, max=40024, per=99.56%, avg=25062.73, stdev=5334.65, samples=166

   iops        : min= 3366, max=10006, avg=6265.57, stdev=1333.67, samples=166

  cpu          : usr=5.13%, sys=18.63%, ctx=202387, majf=0, minf=26

  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%

     issued rwts: total=524625,523951,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):

   READ: bw=24.6MiB/s (25.8MB/s), 24.6MiB/s-24.6MiB/s (25.8MB/s-25.8MB/s), io=2049MiB (2149MB), run=83252-83252msec

  WRITE: bw=24.6MiB/s (25.8MB/s), 24.6MiB/s-24.6MiB/s (25.8MB/s-25.8MB/s), io=2047MiB (2146MB), run=83252-83252msec

Disk stats (read/write):

  vda: ios=523989/526042, merge=0/2218, ticks=3346204/1622070, in_queue=4658999, util=96.06%