[CENTOS] Check server I / O device usage with the Linux iostat command

environment

[root@CENTOS7 ~]# cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)
[root@CENTOS7 ~]#

1. iostat command

You can check the CPU usage and I / O device usage with the ʻiostatcommand. You must havesysstat installed to use the ʻiostat command. See below for installing sysstat. Enable sar command on CentOS (install sysstat)

iostat command format


iostat [option] [Update interval[Number of times]]

Main options for the iostat command

option Contents
-c Output only CPU usage
-d Output only device usage
-g group name Finally, output the total using the group name.
The target device is "-g TOTAL /dev/Can be specified as "sda".
If the target device is ALL, the total of all devices
-h Output in an easy-to-read display format
(Divided into two lines, a device name line and a statistics line)
-j types Display with a persistent name.
ID, LABEL, PATH, UUID can be specified as the type, "-j ID /dev/sda "or"-The target device can be specified like "j ID ALL" (automatically ""-h ”is also specified)
-k Output in KB per second
-m MB output in seconds
-N Displayed by the name of the device mapper
Display the device mapper name when using LVM (Logical Volume Manager)
-p device Specify the device to be displayed.
If you want to output information for each partition, click "-p /dev/Specify as "sda" or "-Specify "p ALL".
In the case of ALL, all devices including unused devices are output (""-If "z" is specified, unused devices will not be output.)
-t Output the measured time
-T Output only total ("-Used with "g")
-x Output extended disk statistics
-y Omit the "statistics (average and total) from system startup" that is displayed first when displaying repeatedly.
-z Do not display devices that are not used during the period to be displayed (devices with zero input / output)

2. How to read the output result of the iostat command

The following is displayed with the ʻiostat` command.

Output result of iostat command


[root@CENTOS7 ~]# iostat
Linux 3.10.0-1062.1.2.el7.x86_64 (CENTOS7)March 02, 2020_x86_64_        (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.18    0.00    0.24    1.22    0.00   98.36

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               2.65        47.22         6.60     140475      19625
dm-0              2.00        43.13         5.91     128305      17577
dm-1              0.03         0.74         0.00       2204          0

[root@CENTOS7 ~]#

2.1 CPU usage items

item Breakdown of CPU usage
%usr Application (user level)
%nice Application (priority specification by nice value)
%system Kernel (system level)
%iowait Idle state (disc I)/Waiting for O request)
%steal Hypervisor processing for other virtual CPUs
%idle Idle state (disc I)/O except waiting for request)

2.2 Device usage items

item Breakdown of device usage
tps I per second/O Number of requests (transfers per second)
kB_read/s Read amount per second (in KB)
kB_wrtn/s Write amount per second (in KB)
kB_read Total read amount (in KB)
kB_wrtn Total write amount (in KB)

2.3 Device usage item (when the -x option is specified)

item Breakdown of device usage
rrqm/s Merged read requests per second
A value indicating the case where multiple read (write) requests for the same device are processed at once.
wrqm/s Merged write requests per second
r/s Read request completed per second (after merging)
w/s Write request completed per second (after merging)
rkB/s Number of read sectors per second
wkB/s Number of write sectors per second
avgrq-sz Average sector size requested at one time (ReQuest)
avgqu-sz I/Average length of O-queue (QUeue)
await Average time to complete a created request
r_await Average time to complete a read request created
w_await Average time to complete a created write request
svctm I issued to the device/Average service time of O request (to be abolished)
%util Device bandwidth utilization

reference

iostat command manual (English)

IOSTAT(1)                     Linux User's Manual                    IOSTAT(1)



NAME
       iostat - Report Central Processing Unit (CPU) statistics and input/out‐
       put statistics for devices and partitions.

SYNOPSIS
       iostat [ -c ] [ -d ] [ -h ] [ -k | -m ] [ -N ] [ -t ] [ -V ] [ -x  ]  [
       -y  ]  [  -z  ]  [  -j { ID | LABEL | PATH | UUID | ... } ] [ [ -T ] -g
       group_name ] [ -p [ device [,...] | ALL ] ] [ device [...] |  ALL  ]  [
       interval [ count ] ]

DESCRIPTION
       The  iostat  command  is used for monitoring system input/output device
       loading by observing the time the devices are  active  in  relation  to
       their average transfer rates. The iostat command generates reports that
       can be used to  change  system  configuration  to  better  balance  the
       input/output load between physical disks.

       The  first  report  generated by the iostat command provides statistics
       concerning the time since the system was booted, unless the  -y  option
       is  used (in this case, this first report is omitted).  Each subsequent
       report covers the time since the previous report.  All  statistics  are
       reported  each time the iostat command is run. The report consists of a
       CPU header row followed by a row of CPU statistics.  On  multiprocessor
       systems,  CPU  statistics  are calculated system-wide as averages among
       all processors. A device header row is displayed followed by a line  of
       statistics for each device that is configured.

       The  interval parameter specifies the amount of time in seconds between
       each report. The first report contains statistics for  the  time  since
       system startup (boot), unless the -y option is used (in this case, this
       report is omitted).  Each subsequent report  contains  statistics  col‐
       lected during the interval since the previous report. The count parame‐
       ter can be specified in conjunction with the interval parameter. If the
       count  parameter is specified, the value of count determines the number
       of reports generated at interval seconds apart. If the interval parame‐
       ter is specified without the count parameter, the iostat command gener‐
       ates reports continuously.


REPORTS
       The iostat command generates two types of reports, the CPU  Utilization
       report and the Device Utilization report.

       CPU Utilization Report
              The first report generated by the iostat command is the CPU Uti‐
              lization Report. For multiprocessor systems, the CPU values  are
              global  averages  among all processors.  The report has the fol‐
              lowing format:

              %user
                     Show the percentage  of  CPU  utilization  that  occurred
                     while executing at the user level (application).

              %nice
                     Show  the  percentage  of  CPU  utilization that occurred
                     while executing at the user level with nice priority.

              %system
                     Show the percentage  of  CPU  utilization  that  occurred
                     while executing at the system level (kernel).

              %iowait
                     Show  the  percentage  of  time that the CPU or CPUs were
                     idle during which the system had an outstanding disk  I/O
                     request.

              %steal
                     Show  the percentage of time spent in involuntary wait by
                     the virtual CPU or CPUs while the hypervisor was  servic‐
                     ing another virtual processor.

              %idle
                     Show  the  percentage  of  time that the CPU or CPUs were
                     idle and the system did not have an outstanding disk  I/O
                     request.

       Device Utilization Report
              The  second report generated by the iostat command is the Device
              Utilization Report. The device report provides statistics  on  a
              per physical device or partition basis. Block devices and parti‐
              tions for which statistics are to be displayed may be entered on
              the  command  line.  If no device nor partition is entered, then
              statistics are displayed for every device used  by  the  system,
              and  providing  that the kernel maintains statistics for it.  If
              the ALL keyword is given on the command  line,  then  statistics
              are  displayed for every device defined by the system, including
              those that have never been used.  Transfer rates are shown in 1K
              blocks  by default, unless the environment variable POSIXLY_COR‐
              RECT is set, in which case 512-byte blocks are used.  The report
              may show the following fields, depending on the flags used:

              Device:
                     This  column  gives  the  device  (or  partition) name as
                     listed in the /dev directory.

              tps
                     Indicate the number of transfers  per  second  that  were
                     issued to the device. A transfer is an I/O request to the
                     device. Multiple logical requests can be combined into  a
                     single  I/O request to the device. A transfer is of inde‐
                     terminate size.

              Blk_read/s (kB_read/s, MB_read/s)
                     Indicate  the  amount  of  data  read  from  the   device
                     expressed  in  a  number of blocks (kilobytes, megabytes)
                     per second. Blocks are equivalent to sectors  and  there‐
                     fore have a size of 512 bytes.

              Blk_wrtn/s (kB_wrtn/s, MB_wrtn/s)
                     Indicate  the  amount  of  data  written  to  the  device
                     expressed in a number of  blocks  (kilobytes,  megabytes)
                     per second.

              Blk_read (kB_read, MB_read)
                     The total number of blocks (kilobytes, megabytes) read.

              Blk_wrtn (kB_wrtn, MB_wrtn)
                     The  total  number of blocks (kilobytes, megabytes) writ‐
                     ten.

              rrqm/s
                     The number of read requests merged per second  that  were
                     queued to the device.

              wrqm/s
                     The  number of write requests merged per second that were
                     queued to the device.

              r/s
                     The number (after merges) of read requests completed  per
                     second for the device.

              w/s
                     The number (after merges) of write requests completed per
                     second for the device.

              rsec/s (rkB/s, rMB/s)
                     The number of sectors (kilobytes,  megabytes)  read  from
                     the device per second.

              wsec/s (wkB/s, wMB/s)
                     The  number  of sectors (kilobytes, megabytes) written to
                     the device per second.

              avgrq-sz
                     The average size (in sectors) of the requests  that  were
                     issued to the device.

              avgqu-sz
                     The average queue length of the requests that were issued
                     to the device.

              await
                     The average  time  (in  milliseconds)  for  I/O  requests
                     issued to the device to be served. This includes the time
                     spent by the requests in queue and the time spent servic‐
                     ing them.

              r_await
                     The  average  time  (in  milliseconds)  for read requests
                     issued to the device to be served. This includes the time
                     spent by the requests in queue and the time spent servic‐
                     ing them.

              w_await
                     The average time (in  milliseconds)  for  write  requests
                     issued to the device to be served. This includes the time
                     spent by the requests in queue and the time spent servic‐
                     ing them.

              svctm
                     The  average  service  time  (in  milliseconds)  for  I/O
                     requests that were issued to the device. Warning! Do  not
                     trust this field any more.  This field will be removed in
                     a future sysstat version.

              %util
                     Percentage of elapsed time during which I/O requests were
                     issued  to  the  device  (bandwidth  utilization  for the
                     device). Device saturation  occurs  when  this  value  is
                     close to 100%.

OPTIONS
       -c     Display the CPU utilization report.

       -d     Display the device utilization report.

       -g group_name { device [...] | ALL }
              Display  statistics  for a group of devices.  The iostat command
              reports statistics for each individual device in the list then a
              line  of global statistics for the group displayed as group_name
              and made up of all the devices in  the  list.  The  ALL  keyword
              means  that all the block devices defined by the system shall be
              included in the group.

       -h     Make the Device Utilization Report easier to read by a human.

       -j { ID | LABEL | PATH | UUID | ... } [ device [...] | ALL ]
              Display persistent device names. Options ID, LABEL, etc. specify
              the  type of the persistent name. These options are not limited,
              only prerequisite is that  directory  with  required  persistent
              names is present in /dev/disk.  Optionally, multiple devices can
              be specified in the chosen persistent name type.   Because  per‐
              sistent  device  names  are  usually  long, option -h is enabled
              implicitly with this option.

       -k     Display statistics in kilobytes per second.

       -m     Display statistics in megabytes per second.

       -N     Display the registered device mapper names for any device mapper
              devices.  Useful for viewing LVM2 statistics.

       -p [ { device [,...] | ALL } ]
              The  -p  option  displays  statistics  for block devices and all
              their partitions that are used by the system.  If a device  name
              is  entered  on the command line, then statistics for it and all
              its partitions are displayed. Last, the  ALL  keyword  indicates
              that  statistics  have to be displayed for all the block devices
              and partitions defined by the system, including those that  have
              never  been  used.  If  option -j is defined before this option,
              devices entered on the command line can be  specified  with  the
              chosen persistent name type.

       -T     This  option must be used with option -g and indicates that only
              global statistics for the group are to  be  displayed,  and  not
              statistics for individual devices in the group.

       -t     Print  the  time for each report displayed. The timestamp format
              may depend on the value of the S_TIME_FORMAT  environment  vari‐
              able (see below).

       -V     Print version number then exit.

       -x     Display extended statistics.

       -y     Omit first report with statistics since system boot, if display‐
              ing multiple records at given interval.

       -z     Tell iostat to omit output for any devices for which  there  was
              no activity during the sample period.


ENVIRONMENT
       The  iostat  command takes into account the following environment vari‐
       ables:


       S_TIME_FORMAT
              If this variable exists and its value is ISO  then  the  current
              locale  will  be  ignored  when  printing the date in the report
              header. The iostat command will use the ISO 8601  format  (YYYY-
              MM-DD)  instead.   The  timestamp  displayed with option -t will
              also be compliant with ISO 8601 format.


       POSIXLY_CORRECT
              When this variable is set, transfer rates are shown in  512-byte
              blocks instead of the default 1K blocks.


EXAMPLES
       iostat
              Display  a  single  history  since  boot  report for all CPU and
              Devices.

       iostat -d 2
              Display a continuous device report at two second intervals.

       iostat -d 2 6
              Display six reports at two second intervals for all devices.

       iostat -x sda sdb 2 6
              Display six reports of extended statistics at two second  inter‐
              vals for devices sda and sdb.

       iostat -p sda 2 6
              Display  six  reports at two second intervals for device sda and
              all its partitions (sda1, etc.)

BUGS
       /proc filesystem must be mounted for iostat to work.

       Kernels older than 2.6.x are no longer supported.

       The average service time (svctm field) value  is  meaningless,  as  I/O
       statistics  are  now  calculated at block level, and we don't know when
       the disk driver starts to process a  request.  For  this  reason,  this
       field will be removed in a future sysstat version.

FILES
       /proc/stat contains system statistics.

       /proc/uptime contains system uptime.

       /proc/diskstats contains disks statistics.

       /sys contains statistics for block devices.

       /proc/self/mountstats contains statistics for network filesystems.

       /dev/disk contains persistent device names.

AUTHOR
       Sebastien Godard (sysstat <at> orange.fr)

SEE ALSO
       sar(1), pidstat(1), mpstat(1), vmstat(8), nfsiostat(1), cifsiostat(1)

       http://pagesperso-orange.fr/sebastien.godard/



Linux                            NOVEMBER 2012                       IOSTAT(1)

iostat command manual (Japanese) (google translate)

IOSTAT(1)                     Linux User's Manual                    IOSTAT(1)



NAME
Reports central processing unit (CPU) statistics and input / output statistics for devices and partitions.

SYNOPSIS
       iostat [ -c ] [ -d ] [ -h ] [ -k | -m ] [ -N ] [ -t ] [ -V ] [ -x  ]  [
       -y  ]  [  -z  ]  [  -j { ID | LABEL | PATH | UUID | ... } ] [ [ -T ] -g
       group_name ] [ -p [ device [,...] | ALL ] ] [ device [...] |  ALL  ]  [
       interval [ count ] ]

DESCRIPTION
The iostat command monitors the amount of time a device has been active in relation to its average transfer rate.
It is used to monitor the load on the system I / O devices.
The iostat command generates a report that can be used to change the system configuration and
Improves the balance of I / O load between physical disks.

The first report generated by the iostat command is-Unless the y option is used
Provides statistics on the time since the system was booted (in this case, this first report is omitted).
Each subsequent report covers the time from the previous report.
All statistics are reported each time the iostat command is executed.
The report consists of a CPU header line followed by a CPU statistics line.
In multiprocessor systems, CPU statistics are calculated system-wide as the average of all processors.
A line of device headers is displayed, followed by a line of statistics for each configured device.

The interval parameter specifies the time, in seconds, between each report.
In the first report,-Unless you use the y option
Contains statistics about the time since the system boots (in this case, this report is omitted).
Each subsequent report contains statistics collected during the interval since the previous report.
The count parameter can be specified in combination with the interval parameter.
If the count parameter is specified, the value of count determines the number of reports generated at interval second intervals.
If the interval parameter is specified without the count parameter, the iostat command produces reports continuously.


REPORTS
The iostat command produces two types of reports: CPU Utilization reports and Device Utilization reports.

       CPU Utilization Report
The first report generated by the iostat command is the CPU utilization report.
For multiprocessor systems, the CPU value is the global average for all processors.
The format of the report is as follows:

              %user
Displays the percentage of CPU usage that occurred during execution at the user level (application).

              %nice
Shows the percentage of CPU usage that occurred during execution at the user level at the nice level.

              %system
Shows the percentage of CPU usage that occurred during execution at the system level (kernel).

              %iowait
Unprocessed disk I in the system/While there is an O request
Shows the percentage of time that one or more CPUs have been idle.

              %steal
While the hypervisor is servicing another virtual processor
Shows the percentage of time that one or more virtual CPUs have inadvertently waited.

              %idle
With one or more CPUs idle,
Unprocessed disk I in the system/O Displays the percentage of time that there was no request.

       Device Utilization Report
The second report generated by the iostat command is the device utilization report.
Device reports provide statistics by physical device or partition.
You can enter the block devices and partitions for which you want to view statistics on the command line.
If no device or partition is entered, statistics for all devices used by the system will be displayed.
Indicates that the kernel holds the statistics.
Specifying the ALL keyword on the command line displays statistics for all system-defined devices.
This includes devices that have never been used.
Environment variable POSIXLY_Transfer rates are displayed in 1K blocks by default unless CORRECT is set.
In that case, a 512-byte block is used.
The report may display the following fields, depending on the flags used:

              Device:
This column is "/Indicates the device (or partition) name listed in the "dev" directory.

              tps
Shows the number of transfers per second issued to the device.
Transfer is I to the device/O request.
Multiple logical requests are a single I to the device/O Can be combined with a request.
The size of the transfer is indefinite.

              Blk_read/s (kB_read/s, MB_read/s)
Shows the amount of data read from the device, expressed in blocks per second (kilobytes, megabytes).
The block is a sector, so it is 512 bytes in size.

              Blk_wrtn/s (kB_wrtn/s, MB_wrtn/s)
Shows the amount of data written to the device, expressed in blocks per second (kilobytes, megabytes).

              Blk_read (kB_read, MB_read)
The total number of blocks read (kilobytes, megabytes).

              Blk_wrtn (kB_wrtn, MB_wrtn)
Total number of blocks written (kilobytes, megabytes).

              rrqm/s
The number of merged read requests per second queued on the device.

              wrqm/s
The number of merged write requests per second queued on the device.

              r/s
The number of read requests completed per second on the device (after merging).

              w/s
The number of completed write requests per second for the device (after merging).

              rsec/s (rkB/s, rMB/s)
The number of sectors (kilobytes, megabytes) read from the device per second.

              wsec/s (wkB/s, wMB/s)
The number of sectors (kilobytes, megabytes) written to the device per second.

              avgrq-sz
The average size of requests issued to the device (in sectors).

              avgqu-sz
The average queue length for requests issued to the device.

              await
I issued to the serviced device/O Average time of request (milliseconds).
This includes the time that requests spend in the queue and the time that they spend on those services.

              r_await
The average time (in milliseconds) for read requests issued to serviced devices.
This includes the time that requests spend in the queue and the time that they spend on those services.

              w_await
The average time (in milliseconds) of a write request issued to a serviced device.
This includes the time that requests spend in the queue and the time that they spend on those services.

              svctm
I issued to the device/O Average service time for requests (milliseconds).
warning! Do not trust this field anymore.
This field will be removed in a future sysstat version.

              %util
                     I/O Percentage of elapsed time the request was issued to the device (device bandwidth utilization).
If this value is close to 100%, device saturation will occur.

OPTIONS
       -c Display the CPU utilization report.

       -d Display the device utilization report.

       -g group_name { device [...] | ALL }
View device group statistics.
The iostat command reports statistics for individual devices in the list and
Then group_Displayed as name and consists of all devices in the list
Report a row of global statistics for the group.
The ALL keyword is used for all block devices defined by the system.
Means to be included in the group.

       -h Make device utilization reports easier for humans to read.

       -j { ID | LABEL | PATH | UUID | ... } [ device [...] | ALL ]
Shows the persistent device name. Option ID, LABEL, etc. specify a persistent name type.
These options are not limited.
The prerequisite is that the directory with the required persistent name is "dev"/It just exists on "disk".
You can optionally specify multiple devices with the persistent name type of your choice.
Persistent device names are usually long, so optional-h is implicitly enabled with this option.

       -k Statistics in kilobytes/Display in seconds.

       -m megabyte/View statistics in seconds.

       -N Device Mapper Displays the registered device mapper name of the device.
Useful for displaying LVM2 statistics.

       -p [ { device [,...] | ALL } ]
              -The p option displays statistics on the block devices used by the system and all its partitions.
Enter the device name on the command line to see statistics for the device and all its partitions.
Finally, the ALL keyword is used with all system-defined block devices.
Indicates that you need to view statistics for partitions (including those that have never been used).
Option before this option-If j is defined
Devices entered on the command line can be specified with the persistent name type of your choice.

       -T This option is optional-Must be used with g,
Only global statistics for the group are displayed,
Indicates that statistics for individual devices in the group will not be displayed.

       -t Print the time for each displayed report.
The format of the time stamp is S_TIME_It may depend on the value of the FORMAT environment variable (see below).

       -Output the V version number and exit.

       -x Display extended statistics.

       -y If you want to view multiple records at specific intervals
Skip the first report that contains statistics since the system booted.

       -Instruct z iostat to omit the output of devices that were inactive during the sample period.


ENVIRONMENT
The iostat command considers the following environment variables:


       S_TIME_FORMAT
If this variable exists and its value is ISO
The current locale is ignored when printing the date in the report header.
The iostat command is instead in ISO 8601 format (YYYY)-MM-DD) is used.
option-The time stamp displayed by t also conforms to the ISO 8601 format.


       POSIXLY_CORRECT
If this variable is set, the transfer rate is displayed in 512 byte blocks instead of the default 1K blocks.


EXAMPLES
       iostat
View a single history since the boot report for all CPUs and devices.

       iostat -d 2
Display continuous device reports at 2-second intervals.

       iostat -d 2 6
View 6 reports at 2-second intervals for all devices.

       iostat -x sda sdb 2 6
View six reports of extended statistics at 2-second intervals for devices sda and sdb.

       iostat -p sda 2 6
View 6 reports every 2 seconds for device sda and all its partitions (such as sda1)

BUGS
For iostat to work,/The "proc" file system must be mounted.

       2.6.Kernels older than x are no longer supported.

       I/The average service time (svctm field) value is now meaningless because O statistics are now calculated at the block level.
I don't know when the disk driver will start processing the request.
For this reason, this field will be removed in a future sysstat version.

FILES
       /proc/stat contains system statistics.

       /proc/uptime contains system uptime.

       /proc/diskstats contains disks statistics.

       /sys contains statistics for block devices.

       /proc/self/mountstats contains statistics for network filesystems.

       /dev/disk contains persistent device names.

AUTHOR
       Sebastien Godard (sysstat <at> orange.fr)

SEE ALSO
       sar(1), pidstat(1), mpstat(1), vmstat(8), nfsiostat(1), cifsiostat(1)

       http://pagesperso-orange.fr/sebastien.godard/



Linux                            NOVEMBER 2012                       IOSTAT(1)

that's all

Recommended Posts

Check server I / O device usage with the Linux iostat command
Check the operating status of the server with the Linux top command
Check the HTTP status code response with the curl command (#Linux #Shell)
Adjust file permissions with the Linux command chmod
[Linux] Execute git pull regularly with the crontab command
I tried to rewrite the WEB server of the normal Linux programming 1st edition with C ++ 14
linux sar command CPU usage
I examined the device tree
Check the code with flake8
Linux PC spec check command
[Linux] I tried to summarize the command of resource confirmation system
I tried sending an email from the Sakura server with flask-mail
Check the memory protection of linux kerne with the code for ARM
I want to check the position of my face with OpenCV!
I used the worldcup command to check the outcome of the World Cup.
Every time I draw a graph with python, I check it, so I will summarize only the simplest usage
[Linux] OS recovery with restore command
LINUX command [wc edition] Usage example
[linux] kill command to kill the process
I liked the tweet with python. ..
Using cgo with the go command
Hit the top command with htop
I investigated the device tree Overlay
Replace the directory name and the file name in the directory together with a Linux command.
I tried Linux (CentOS 7) life and death monitoring (Ping) with monitoring server Zabbix
I get a Python No module named'encodings' error with the aws command
The --security option cannot be used with the dnf command on CentOS Linux
VS Code freezes & PC crashes when I start the server with go
[Linux] Why do I, an infrastructure engineer, not use the "hostname" command?
I tried to control the network bandwidth and delay with the tc command