SAR 10 USES
1. CPU Usage of ALL CPUs (sar -u)
This gives the cumulative real-time CPU usage of all CPUs. “1 3″ reports for every 1 seconds a total of 3 times. Most likely you’ll focus on the last field “%idle” to see the cpu load.<span style="font-family:'courier new', courier;">$ sar -u 1 3
Linux 2.6.18-194.el5PAE (dev-db) 03/26/2011 _i686_ (8 CPU)
01:27:32 PM CPU %user %nice %system %iowait %steal %idle
01:27:33 PM all 0.00 0.00 0.00 0.00 0.00 100.00
01:27:34 PM all 0.25 0.00 0.25 0.00 0.00 99.50
01:27:35 PM all 0.75 0.00 0.25 0.00 0.00 99.00
Average: all 0.33 0.00 0.17 0.00 0.00 99.50</span>
Following are few variations:- sar -u Displays CPU usage for the current day that was collected until that point.
- sar -u 1 3 Displays real time CPU usage every 1 second for 3 times.
- sar -u ALL Same as “sar -u” but displays additional fields.
- sar -u ALL 1 3 Same as “sar -u 1 3″ but displays additional fields.
- sar -u -f /var/log/sa/sa10 Displays CPU usage for the 10day of the month from the sa10 file.
2. CPU Usage of Individual CPU or Core (sar -P)
If you have 4 Cores on the machine and would like to see what the individual cores are doing, do the following.“-P ALL” indicates that it should displays statistics for ALL the individual Cores.In the following example under “CPU” column 0, 1, 2, and 3 indicates the corresponding CPU core numbers.<span style="font-family:'courier new', courier;">$ sar -P ALL 1 1
Linux 2.6.18-194.el5PAE (dev-db) 03/26/2011 _i686_ (8 CPU)
01:34:12 PM CPU %user %nice %system %iowait %steal %idle
01:34:13 PM all 11.69 0.00 4.71 0.69 0.00 82.90
01:34:13 PM 0 35.00 0.00 6.00 0.00 0.00 59.00
01:34:13 PM 1 22.00 0.00 5.00 0.00 0.00 73.00
01:34:13 PM 2 3.00 0.00 1.00 0.00 0.00 96.00
01:34:13 PM 3 0.00 0.00 0.00 0.00 0.00 100.00</span>
“-P 1″ indicates that it should displays statistics only for the 2nd Core. (Note that Core number starts from 0).<span style="font-family:'courier new', courier;">$ sar -P 1 1 1
Linux 2.6.18-194.el5PAE (dev-db) 03/26/2011 _i686_ (8 CPU)
01:36:25 PM CPU %user %nice %system %iowait %steal %idle
01:36:26 PM 1 8.08 0.00 2.02 1.01 0.00 88.89</span>
Following are few variations:- sar -P ALL Displays CPU usage broken down by all cores for the current day.
- sar -P ALL 1 3 Displays real time CPU usage for ALL cores every 1 second for 3 times (broken down by all cores).
- sar -P 1 Displays CPU usage for core number 1 for the current day.
- sar -P 1 1 3 Displays real time CPU usage for core number 1, every 1 second for 3 times.
- sar -P ALL -f /var/log/sa/sa10 Displays CPU usage broken down by all cores for the 10day day of the month from sa10 file.
3. Memory Free and Used (sar -r)
This reports the memory statistics. “1 3″ reports for every 1 seconds a total of 3 times. Most likely you’ll focus on “kbmemfree” and “kbmemused” for free and used memory.<span style="font-family:'courier new', courier;">$ sar -r 1 3
Linux 2.6.18-194.el5PAE (dev-db) 03/26/2011 _i686_ (8 CPU)
07:28:06 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact
07:28:07 AM 6209248 2097432 25.25 189024 1796544 141372 0.85 1921060 88204
07:28:08 AM 6209248 2097432 25.25 189024 1796544 141372 0.85 1921060 88204
07:28:09 AM 6209248 2097432 25.25 189024 1796544 141372 0.85 1921060 88204
Average: 6209248 2097432 25.25 189024 1796544 141372 0.85 1921060 88204</span>
Following are few variations:- sar -r
- sar -r 1 3
- sar -r -f /var/log/sa/sa10
4. Swap Space Used (sar -S)
This reports the swap statistics. “1 3″ reports for every 1 seconds a total of 3 times. If the “kbswpused” and “%swpused” are at 0, then your system is not swapping.<span style="font-family:'courier new', courier;">$ sar -S 1 3
Linux 2.6.18-194.el5PAE (dev-db) 03/26/2011 _i686_ (8 CPU)
07:31:06 AM kbswpfree kbswpused %swpused kbswpcad %swpcad
07:31:07 AM 8385920 0 0.00 0 0.00
07:31:08 AM 8385920 0 0.00 0 0.00
07:31:09 AM 8385920 0 0.00 0 0.00
Average: 8385920 0 0.00 0 0.00</span>
Following are few variations:- sar -S
- sar -S 1 3
- sar -S -f /var/log/sa/sa10
Notes:- Use “sar -R” to identify number of memory pages freed, used, and cached per second by the system.
- Use “sar -H” to identify the hugepages (in KB) that are used and available.
- Use “sar -B” to generate paging statistics. i.e Number of KB paged in (and out) from disk per second.
- Use “sar -W” to generate page swap statistics. i.e Page swap in (and out) per second.
5. Overall I/O Activities (sar -b)
This reports I/O statistics. “1 3″ reports for every 1 seconds a total of 3 times.Following fields are displays in the example below.- tps – Transactions per second (this includes both read and write)
- rtps – Read transactions per second
- wtps – Write transactions per second
- bread/s – Bytes read per second
- bwrtn/s – Bytes written per second
<span style="font-family:'courier new', courier;">$ sar -b 1 3
Linux 2.6.18-194.el5PAE (dev-db) 03/26/2011 _i686_ (8 CPU)
01:56:28 PM tps rtps wtps bread/s bwrtn/s
01:56:29 PM 346.00 264.00 82.00 2208.00 768.00
01:56:30 PM 100.00 36.00 64.00 304.00 816.00
01:56:31 PM 282.83 32.32 250.51 258.59 2537.37
Average: 242.81 111.04 131.77 925.75 1369.90</span>
Following are few variations:- sar -b
- sar -b 1 3
- sar -b -f /var/log/sa/sa10
Note: Use “sar -v” to display number of inode handlers, file handlers, and pseudo-terminals used by the system.
6. Individual Block Device I/O Activities (sar -d)
To identify the activities by the individual block devices (i.e a specific mount point, or LUN, or partition), use “sar -d”<span style="font-family:'courier new', courier;">$ sar -d 1 1
Linux 2.6.18-194.el5PAE (dev-db) 03/26/2011 _i686_ (8 CPU)
01:59:45 PM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
01:59:46 PM dev8-0 1.01 0.00 0.00 0.00 0.00 4.00 1.00 0.10
01:59:46 PM dev8-1 1.01 0.00 0.00 0.00 0.00 4.00 1.00 0.10
01:59:46 PM dev120-64 3.03 64.65 0.00 21.33 0.03 9.33 5.33 1.62
01:59:46 PM dev120-65 3.03 64.65 0.00 21.33 0.03 9.33 5.33 1.62
01:59:46 PM dev120-0 8.08 0.00 105.05 13.00 0.00 0.38 0.38 0.30
01:59:46 PM dev120-1 8.08 0.00 105.05 13.00 0.00 0.38 0.38 0.30
01:59:46 PM dev120-96 1.01 8.08 0.00 8.00 0.01 9.00 9.00 0.91
01:59:46 PM dev120-97 1.01 8.08 0.00 8.00 0.01 9.00 9.00 0.91</span>
In the above example “DEV” indicates the specific block device.For example: “dev53-1″ means a block device with 53 as major number, and 1 as minor number.The device name (DEV column) can display the actual device name (for example: sda, sda1, sdb1 etc.,), if you use the -p option (pretty print) as shown below.<span style="font-family:'courier new', courier;">$ sar -p -d 1 1
Linux 2.6.18-194.el5PAE (dev-db) 03/26/2011 _i686_ (8 CPU)
01:59:45 PM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
01:59:46 PM sda 1.01 0.00 0.00 0.00 0.00 4.00 1.00 0.10
01:59:46 PM sda1 1.01 0.00 0.00 0.00 0.00 4.00 1.00 0.10
01:59:46 PM sdb1 3.03 64.65 0.00 21.33 0.03 9.33 5.33 1.62
01:59:46 PM sdc1 3.03 64.65 0.00 21.33 0.03 9.33 5.33 1.62
01:59:46 PM sde1 8.08 0.00 105.05 13.00 0.00 0.38 0.38 0.30
01:59:46 PM sdf1 8.08 0.00 105.05 13.00 0.00 0.38 0.38 0.30
01:59:46 PM sda2 1.01 8.08 0.00 8.00 0.01 9.00 9.00 0.91
01:59:46 PM sdb2 1.01 8.08 0.00 8.00 0.01 9.00 9.00 0.91</span>
Following are few variations:- sar -d
- sar -d 1 3
- sar -d -f /var/log/sa/sa10
- sar -p -d
7. Display context switch per second (sar -w)
This reports the total number of processes created per second, and total number of context switches per second. “1 3″ reports for every 1 seconds a total of 3 times.<span style="font-family:'courier new', courier;">$ sar -w 1 3
Linux 2.6.18-194.el5PAE (dev-db) 03/26/2011 _i686_ (8 CPU)
08:32:24 AM proc/s cswch/s
08:32:25 AM 3.00 53.00
08:32:26 AM 4.00 61.39
08:32:27 AM 2.00 57.00</span>
Following are few variations:- sar -w
- sar -w 1 3
- sar -w -f /var/log/sa/sa10
8. Reports run queue and load average (sar -q)
This reports the run queue size and load average of last 1 minute, 5 minutes, and 15 minutes. “1 3″ reports for every 1 seconds a total of 3 times.<span style="font-family:'courier new', courier;">$ sar -q 1 3
Linux 2.6.18-194.el5PAE (dev-db) 03/26/2011 _i686_ (8 CPU)
06:28:53 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked
06:28:54 AM 0 230 2.00 3.00 5.00 0
06:28:55 AM 2 210 2.01 3.15 5.15 0
06:28:56 AM 2 230 2.12 3.12 5.12 0
Average: 3 230 3.12 3.12 5.12 0</span>
Note: The “blocked” column displays the number of tasks that are currently blocked and waiting for I/O operation to complete.Following are few variations:- sar -q
- sar -q 1 3
- sar -q -f /var/log/sa/sa10
9. Report network statistics (sar -n)
This reports various network statistics. For example: number of packets received (transmitted) through the network card, statistics of packet failure etc.,. “1 3″ reports for every 1 seconds a total of 3 times.<span style="font-family:'courier new', courier;">sar -n KEYWORD</span>
KEYWORD can be one of the following:- DEV – Displays network devices vital statistics for eth0, eth1, etc.,
- EDEV – Display network device failure statistics
- NFS – Displays NFS client activities
- NFSD – Displays NFS server activities
- SOCK – Displays sockets in use for IPv4
- IP – Displays IPv4 network traffic
- EIP – Displays IPv4 network errors
- ICMP – Displays ICMPv4 network traffic
- EICMP – Displays ICMPv4 network errors
- TCP – Displays TCPv4 network traffic
- ETCP – Displays TCPv4 network errors
- UDP – Displays UDPv4 network traffic
- SOCK6, IP6, EIP6, ICMP6, UDP6 are for IPv6
- ALL – This displays all of the above information. The output will be very long.
<span style="font-family:'courier new', courier;">$ sar -n DEV 1 1
Linux 2.6.18-194.el5PAE (dev-db) 03/26/2011 _i686_ (8 CPU)
01:11:13 PM IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/s rxmcst/s
01:11:14 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:11:14 PM eth0 342.57 342.57 93923.76 141773.27 0.00 0.00 0.00
01:11:14 PM eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00</span>
10. Report Sar Data Using Start Time (sar -s)
When you view historic sar data from the /var/log/sa/saXX file using “sar -f” option, it displays all the sar data for that specific day starting from 12:00 a.m for that day.Using “-s hh:mi:ss” option, you can specify the start time. For example, if you specify “sar -s 10:00:00″, it will display the sar data starting from 10 a.m (instead of starting from midnight) as shown below.You can combine -s option with other sar option.For example, to report the load average on 26th of this month starting from 10 a.m in the morning, combine the -q and -s option as shown below.<span style="font-family:'courier new', courier;">$ sar -q -f /var/log/sa/sa23 -s 10:00:01
Linux 2.6.18-194.el5PAE (dev-db) 03/26/2011 _i686_ (8 CPU)
10:00:01 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked
10:10:01 AM 0 127 2.00 3.00 5.00 0
10:20:01 AM 0 127 2.00 3.00 5.00 0
...
11:20:01 AM 0 127 5.00 3.00 3.00 0
12:00:01 PM 0 127 4.00 2.00 1.00 0</span>
There is no option to limit the end-time. You just have to get creative and use head command as shown below.For example, starting from 10 a.m, if you want to see 7 entries, you have to pipe the above output to “head -n 10″.<span style="font-family:'courier new', courier;">$ sar -q -f /var/log/sa/sa23 -s 10:00:01 | head -n 10
Linux 2.6.18-194.el5PAE (dev-db) 03/26/2011 _i686_ (8 CPU)
10:00:01 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked
10:10:01 AM 0 127 2.00 3.00 5.00 0
10:20:01 AM 0 127 2.00 3.00 5.00 0
10:30:01 AM 0 127 3.00 5.00 2.00 0
10:40:01 AM 0 127 4.00 2.00 1.00 2
10:50:01 AM 0 127 3.00 5.00 5.00 0
11:00:01 AM 0 127 2.00 1.00 6.00 0
11:10:01 AM 0 127 1.00 3.00 7.00 2</span>
There is lot more to cover in Linux performance monitoring and tuning. We are only getting started. More articles to come in the performance series.SAR - Syntax and Definations
SYNOPSIS
sar [ -A ] [ -b ] [ -B ] [ -C ] [ -d ] [ -h ] [ -i interval ] [ -m ] [
-p ] [ -q ] [ -r ] [ -R ] [ -S ] [ -t ] [ -u [ ALL ] ] [ -v ] [ -V ] [
-w ] [ -W ] [ -y ] [ -n { keyword [,...] | ALL } ] [ -I { int [,...] |
SUM | ALL | XALL } ] [ -P { cpu [,...] | ALL } ] [ -o [ filename ] | -f
[ filename ] ] [ -s [ hh:mm:ss ] ] [ -e [ hh:mm:ss ] ] [ interval [
count ] ]
DESCRIPTION
The sar command writes to standard output the contents of selected cumulative activity counters in the operating system. The accounting
system, based on the values in the count and interval parameters, writes information the specified number of times spaced at the speci-
fied intervals in seconds. If the interval parameter is set to zero, the sar command displays the average statistics for the time since the
system was started. If the interval parameter is specified without the count parameter, then reports are generated continuously. The col-
lected data can also be saved in the file specified by the -o filename flag, in addition to being displayed onto the screen. If filename is
omitted, sar uses the standard system activity daily data file, the /var/log/sysstat/sadd file, where the dd parameter indicates the cur-
rent day. By default all the data available from the kernel are saved in the data file.
The sar command extracts and writes to standard output records previously saved in a file. This file can be either the one specified by the
-f flag or, by default, the standard system activity daily data file.
Without the -P flag, the sar command reports system-wide (global among all processors) statistics, which are calculated as averages for values
expressed as percentages, and as sums otherwise. If the -P flag is given, the sar command reports activity which relates to the specified
processor or processors. If -P ALL is given, the sar command reports statistics for each individual processor and global statistics among
all processors.
You can select information about specific system activities using flags. Not specifying any flags selects only CPU activity. Specifying
the -A flag is equivalent to specifying -bBdqrRSvwWy -I SUM -I XALL -n ALL -u ALL -P ALL.
The default version of the sar command (CPU utilization report) might be one of the first facilities the user runs to begin system activity
investigation, because it monitors major system resources. If CPU utilization is near 100 percent (user + nice + system), the workload sam-
pled is CPU-bound.
If multiple samples and multiple reports are desired, it is convenient to specify an output file for the sar command. Run the sar command as
a background process. The syntax for this is:
sar -o datafile interval count >/dev/null 2>&1 &
-A This is equivalent to specifying -bBdqrRSuvwWy -I SUM -I XALL -n
ALL -u ALL -P ALL.
-b Report I/O and transfer rate statistics. The following values are displayed:
tps Total number of transfers per second that were issued to physical devices. A transfer is an I/O request to a
physical device. Multiple logical requests can be combined into a single I/O request to the device. A trans-
fer is of indeterminate size.
rtps Total number of read requests per second issued to physical devices.
wtps Total number of write requests per second issued to physical devices.
bread/s Total amount of data read from the devices in blocks per second. Blocks are equivalent to sectors with 2.4 ker-
nels and newer and therefore have a size of 512 bytes. With older kernels, a block is of indeterminate size.
bwrtn/s Total amount of data written to devices in blocks per second.
-B Report paging statistics. Some of the metrics below are available only with post 2.5 kernels. The following values are displayed:
pgpgin/s Total number of kilobytes the system paged in from disk per second. Note: With old kernels (2.2.x) this value is
a number of blocks per second (and not kilobytes).
pgpgout/s Total number of kilobytes the system paged out to disk per second. Note: With old kernels (2.2.x) this value is
a number of blocks per second (and not kilobytes).
fault/s Number of page faults (major + minor) made by the system per second. This is not a count of page faults that gen-
erate I/O, because some page faults can be resolved without I/O.
majflt/s Number of major faults the system has made per second, those which have required loading a memory page from
pgsteal/s Number of pages the system has reclaimed from cache (pagecache and swapcache) per second to satisfy its mem-
ory demands.
%vmeff Calculated as pgsteal / pgscan, this is a metric of the efficiency of page reclaim. If it is near 100% then
almost every page coming off the tail of the inactive list is being reaped. If it gets too low (e.g. less than
30%) then the virtual memory is having some difficulty. This field is displayed as zero if no pages have been scanned during the interval of time.
-C When reading data from a file, tell sar to display comments that have been inserted by sadc.
-d Report activity for each block device (kernels 2.4 and newer only). When data is displayed, the device specification dev m-n
is generally used ( DEV column). m is the major number of the device. With recent kernels (post 2.5), n is the minor number
of the device, but is only a sequence number with pre 2.5 kernels. Device names may also be pretty-printed if option -p is
used (see below). Values for fields avgqu-sz, await, svctm and %util may be unavailable and displayed as 0.00 with some 2.4
kernels. Note that disk activity depends on sadc options "-S DISK" and "-S XDISK" to be collected. The following values are
displayed:
tps Indicate the number of transfers per second that were issued to the device. Multiple logical requests can be
combined into a single I/O request to the device. A transfer is of indeterminate size.
rd_sec/s Number of sectors read from the device. The size of a sector is 512 bytes.
wr_sec/s Number of sectors written to the device. The size of a sector is 512 bytes.
avgrq-sz The average size (in sectors) of the requests that were issued to the device.
avgqu-sz The average queue length of the requests that were issued to the device.
await The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time
-e [ hh:mm:ss ]
Set the ending time of the report. The default ending time is 18:00:00. Hours must be given in 24-hour format. This option
can be used when data are read from or written to a file (options -f or -o ).
-f [ filename ]
Extract records from filename (created by the -o filename flag). The default value of the filename parameter is the current daily
data file, the /var/log/sysstat/sadd file. The -f option is exclusive of the -o option.
-h Display a short help message then exit.
-i interval
Select data records at seconds as close as possible to the number specified by the interval parameter.
-I { int [,...] | SUM | ALL | XALL }
Report statistics for a given interrupt. int is the interrupt number. Specifying multiple -I int parameters on the command
line will look at multiple independent interrupts. The SUM keyword indicates that the total number of interrupts received per
second is to be displayed. The ALL keyword indicates that statistics from the first 16 interrupts are to be reported, whereas
the XALL keyword indicates that statistics from all interrupts, including potential APIC interrupt sources, are to be reported.
Note that interrupt statistics depend on sadc option "-S INT" to be collected.
-m Report power management statistics. Note that these statistics depend on sadc option "-S POWER" to be collected. The following
value is displayed:
MHz
CPU clock frequency in MHz.
-n { keyword [,...] | ALL }
Report network statistics.
Possible keywords are DEV, EDEV, NFS, NFSD, SOCK, IP, EIP, ICMP,
EICMP, TCP, ETCP, UDP, SOCK6, IP6, EIP6, ICMP6, EICMP6 and UDP6.
With the DEV keyword, statistics from the network devices are reported. The following values are displayed:
IFACE Name of the network interface for which statistics are reported.
rxpck/s Total number of packets received per second.
txcmp/s Number of compressed packets transmitted per second.
rxmcst/s Number of multicast packets received per second.
With the EDEV keyword, statistics on failures (errors) from the network devices are reported. The following values are dis-
played:
IFACE Name of the network interface for which statistics are reported.
rxerr/s Total number of bad packets received per second.
txerr/s Total number of errors that happened per second while transmitting packets.
coll/s Number of collisions that happened per second while transmitting packets.
rxdrop/s Number of received packets dropped per second because of a lack of space in linux buffers.
txdrop/s Number of transmitted packets dropped per second because of a lack of space in linux buffers.
txcarr/s Number of carrier-errors that happened per second while transmitting packets.
rxfram/s Number of frame alignment errors that happened per second on received packets.
rxfifo/s Number of FIFO overrun errors that happened per second on received packets.
txfifo/s Number of FIFO overrun errors that happened per second on transmitted packets.
With the NFS keyword, statistics about NFS client activity are
reported. The following values are displayed:
call/saccess/s Number of 'access' RPC calls made per second.
getatt/s Number of 'getattr' RPC calls made per second.
With the NFSD keyword, statistics about NFS server activity are
reported. The following values are displayed:
scall/s Number of RPC requests received per second.
badcall/s Number of bad RPC requests received per second, those
whose processing generated an error.
packet/s Number of network packets received per second.
udp/s Number of UDP packets received per second.
tcp/s Number of TCP packets received per second.
hit/s Number of reply cache hits per second.
miss/s Number of reply cache misses per second.
sread/s Number of 'read' RPC calls received per second.
swrite/s Number of 'write' RPC calls received per second.
saccess/s Number of 'access' RPC calls received per second.
sgetatt/s Number of 'getattr' RPC calls received per second.
With the SOCK keyword, statistics on sockets in use are reported
(IPv4). The following values are displayed:
totsck Total number of sockets used by the system.
tcpsck Number of TCP sockets currently in use.
udpsck reported. Note that IPv4 statistics depend on sadc option "-S SNMP" to be collected. The following values are displayed (for-
mal SNMP names between square brackets):
irec/s The total number of input datagrams received from interfaces per second, including those received in error
[ipInReceives].
fwddgm/s The number of input datagrams per second, for which this entity was not their final IP destination, as a result of
which an attempt was made to find a route to forward them to that final destination [ipForwDatagrams].
idel/s The total number of input datagrams successfully delivered per second to IP user-protocols (including ICMP)
[ipInDelivers].
orq/s The total number of IP datagrams which local IP user-protocols (including ICMP) supplied per second to IP in
requests for transmission [ipOutRequests]. Note that this counter does not include any datagrams counted in
fwddgm/s.
asmrq/s The number of IP fragments received per second which needed to be reassembled at this entity [ipReasmReqds].
asmok/s The number of IP datagrams successfully re-assembled per second [ipReasmOKs].
fragok/s The number of IP datagrams that have been successfully fragmented at this entity per second [ipFragOKs].
fragcrt/s The number of IP datagram fragments that have been generated per second as a result of fragmentation at this
entity [ipFragCreates].
With the EIP keyword, statistics about IPv4 network errors are reported. Note that IPv4 statistics depend on sadc option "-S
SNMP" to be collected. The following values are displayed (formal SNMP names between square brackets):
ihdrerr/s The number of input datagrams discarded per second due to errors in their IP headers, including bad checksums, ver-
sion number mismatch, other format errors, time-to-live exceeded, errors discovered in processing their IP
iukwnpr/s The number of locally-addressed datagrams received successfully but discarded per second because of an unknown
or unsupported protocol [ipInUnknownProtos].
idisc/s The number of input IP datagrams per second for which no problems were encountered to prevent their continued pro-
cessing, but which were discarded (e.g., for lack of buffer space) [ipInDiscards]. Note that this counter does
not include any datagrams discarded while awaiting re-assembly.
odisc/s The number of output IP datagrams per second for which no problem was encountered to prevent their transmission to
their destination, but which were discarded (e.g., for lack of buffer space) [ipOutDiscards]. Note that this
counter would include datagrams counted in fwddgm/s if any such packets met this (discretionary) discard crite-
rion.
onort/s The number of IP datagrams discarded per second because no route could be found to transmit them to their desti-
nation [ipOutNoRoutes]. Note that this counter includes any packets counted in fwddgm/s which meet this
'no-route' criterion. Note that this includes any datagrams which a host cannot route because all of its
default routers are down.
asmf/s The number of failures detected per second by the IP re-assembly algorithm (for whatever reason: timed out,
errors, etc) [ipReasmFails]. Note that this is not necessarily a count of discarded IP fragments since some
algorithms can lose track of the number of fragments by combining them as they are received.
fragf/s The number of IP datagrams that have been discarded per second because they needed to be fragmented at this
entity but could not be, e.g., because their Don't Fragment flag was set [ipFragFails].
With the ICMP keyword, statistics about ICMPv4 network traffic are reported. Note that ICMPv4 statistics depend on sadc option
"-S SNMP" to be collected. The following values are displayed (formal SNMP names between square brackets):
imsg/s The total number of ICMP messages which the entity received per second [icmpInMsgs]. Note that this counter
The number of ICMP Echo Reply messages received per second [icmpInEchoReps].
oech/s The number of ICMP Echo (request) messages sent per second [icmpOutEchos].
oechr/s The number of ICMP Echo Reply messages sent per second [icmpOutEchoReps].
itm/s The number of ICMP Timestamp (request) messages received per second [icmpInTimestamps].
itmr/s The number of ICMP Timestamp Reply messages received per second [icmpInTimestampReps].
otm/s The number of ICMP Timestamp (request) messages sent per second [icmpOutTimestamps].
otmr/s The number of ICMP Timestamp Reply messages sent per second [icmpOutTimestampReps].
iadrmk/s The number of ICMP Address Mask Request messages received per second [icmpInAddrMasks].
iadrmkr/s The number of ICMP Address Mask Reply messages received per second [icmpInAddrMaskReps].
oadrmk/s The number of ICMP Address Mask Request messages sent per second [icmpOutAddrMasks].
oadrmkr/s The number of ICMP Address Mask Reply messages sent per second [icmpOutAddrMaskReps].
With the EICMP keyword, statistics about ICMPv4 error messages
are reported. Note that ICMPv4 statistics depend on sadc option
"-S SNMP" to be collected. The following values are displayed
(formal SNMP names between square brackets):
ierr/s
The number of ICMP messages per second which the entity
received but determined as having ICMP-specific errors
(bad ICMP checksums, bad length, etc.) [icmpInErrors].
oerr/s
itmex/s
The number of ICMP Time Exceeded messages received per
second [icmpInTimeExcds].
otmex/s
The number of ICMP Time Exceeded messages sent per second
[icmpOutTimeExcds].
iparmpb/s
The number of ICMP Parameter Problem messages received
per second [icmpInParmProbs].
oparmpb/s
The number of ICMP Parameter Problem messages sent per
second [icmpOutParmProbs].
isrcq/s
The number of ICMP Source Quench messages received per
second [icmpInSrcQuenchs].
osrcq/s
The number of ICMP Source Quench messages sent per second
[icmpOutSrcQuenchs].
iredir/s
The number of ICMP Redirect messages received per second
[icmpInRedirects].
oredir/s
The number of ICMP Redirect messages sent per second
[icmpOutRedirects].
With the TCP keyword, statistics about TCPv4 network traffic are
reported. Note that TCPv4 statistics depend on sadc option "-S
SNMP" to be collected. The following values are displayed (for-
mal SNMP names between square brackets):
active/s
The number of times TCP connections have made a direct
transition to the SYN-SENT state from the CLOSED state
per second [tcpActiveOpens].
passive/s
The number of times TCP connections have made a direct
transition to the SYN-RCVD state from the LISTEN state
per second [tcpPassiveOpens].
iseg/s
The total number of segments received per second, including those received in error [tcpInSegs]. This count
includes segments received on currently established connections.
oseg/s
SYN-SENT state or the SYN-RCVD state, plus the number of times per second TCP connections have made a direct tran-
sition to the LISTEN state from the SYN-RCVD state [tcpAttemptFails].
estres/s
The number of times per second TCP connections have made a direct transition to the CLOSED state from either the
ESTABLISHED state or the CLOSE-WAIT state [tcpEstabResets].
retrans/s
The total number of segments retransmitted per second that is, the number of TCP segments transmitted contain-
ing one or more previously transmitted octets [tcpRetransSegs].
isegerr/s
The total number of segments received in error (e.g., bad TCP checksums) per second [tcpInErrs].
orsts/s
The number of TCP segments sent per second containing the RST flag [tcpOutRsts].
With the UDP keyword, statistics about UDPv4 network traffic are reported. Note that UDPv4 statistics depend on sadc option "-S
SNMP" to be collected. The following values are displayed (formal SNMP names between square brackets):
idgm/s
The total number of UDP datagrams delivered per second to UDP users [udpInDatagrams].
odgm/s
The total number of UDP datagrams sent per second from this entity [udpOutDatagrams].
noport/s
The total number of received UDP datagrams per second for which there was no application at the destination port
[udpNoPorts].
idgmerr/s
The number of received UDP datagrams per second that could not be delivered for reasons other than the lack of
an application at the destination port [udpInErrors].
With the SOCK6 keyword, statistics on sockets in use are reported (IPv6). Note that IPv6 statistics depend on sadc
option "-S IPV6" to be collected. The following values are displayed:
tcp6sck
reported. Note that IPv6 statistics depend on sadc option "-S IPV6" to be collected. The following values are displayed (for-
mal SNMP names between square brackets):
irec6/s
The total number of input datagrams received from interfaces per second, including those received in error
[ipv6IfStatsInReceives].
fwddgm6/s
The number of output datagrams per second which this entity received and forwarded to their final destinations
[ipv6IfStatsOutForwDatagrams].
idel6/s
The total number of datagrams successfully delivered per second to IPv6 user-protocols (including ICMP) [ipv6IfS-
tatsInDelivers].
orq6/s
The total number of IPv6 datagrams which local IPv6 user-protocols (including ICMP) supplied per second to
IPv6 in requests for transmission [ipv6IfStatsOutRequests]. Note that this counter does not include any
datagrams counted in fwddgm6/s.
asmrq6/s
The number of IPv6 fragments received per second which needed to be reassembled at this interface [ipv6IfStat-
sReasmReqds].
asmok6/s
The number of IPv6 datagrams successfully reassembled per second [ipv6IfStatsReasmOKs].
imcpck6/s
The number of multicast packets received per second by the interface [ipv6IfStatsInMcastPkts].
omcpck6/s
The number of multicast packets transmitted per second by the interface [ipv6IfStatsOutMcastPkts].
fragok6/s
The number of IPv6 datagrams that have been successfully fragmented at this output interface per second [ipv6IfS-
tatsOutFragOKs].
fragcr6/s
The number of output datagram fragments that have been generated per second as a result of fragmentation at this
output interface [ipv6IfStatsOutFragCreates].
With the EIP6 keyword, statistics about IPv6 network errors are
The number of input datagrams discarded per second because the IPv6 address in their IPv6 header's destina-
tion field was not a valid address to be received at this entity. This count includes invalid addresses (e.g., ::0)
and unsupported addresses (e.g., addresses with unallocated prefixes). For entities which are not IPv6 routers
and therefore do not forward datagrams, this counter includes datagrams discarded because the destination
address was not a local address [ipv6IfStatsInAddrErrors].
iukwnp6/s
The number of locally-addressed datagrams received successfully but discarded per second because of an unknown
or unsupported protocol [ipv6IfStatsInUnknownProtos].
i2big6/s
The number of input datagrams that could not be forwarded per second because their size exceeded the link MTU of
outgoing interface [ipv6IfStatsInTooBigErrors].
idisc6/s
The number of input IPv6 datagrams per second for which no problems were encountered to prevent their continued
processing, but which were discarded (e.g., for lack of buffer space) [ipv6IfStatsInDiscards]. Note that this
counter does not include any datagrams discarded while awaiting re-assembly.
odisc6/s
The number of output IPv6 datagrams per second for which no problem was encountered to prevent their transmission
to their destination, but which were discarded (e.g., for lack of buffer space) [ipv6IfStatsOutDiscards]. Note that
this counter would include datagrams counted in fwddgm6/s if any such packets met this (discretionary) discard cri-
terion.
inort6/s
The number of input datagrams discarded per second because no route could be found to transmit them to their
destination [ipv6IfStatsInNoRoutes].
onort6/s
The number of locally generated IP datagrams discarded per second because no route could be found to transmit
them to their destination [unknown formal SNMP name].
asmf6/s
The number of failures detected per second by the IPv6 re-assembly algorithm (for whatever reason: timed out,
errors, etc.) [ipv6IfStatsReasmFails]. Note that this is not necessarily a count of discarded IPv6 fragments since
some algorithms can lose track of the number of fragments
With the ICMP6 keyword, statistics about ICMPv6 network traffic are reported. Note that ICMPv6 statistics depend on sadc option
"-S IPV6" to be collected. The following values are displayed (formal SNMP names between square brackets):
imsg6/s
The total number of ICMP messages received by the interface per second which includes all those counted by
ierr6/s [ipv6IfIcmpInMsgs].
omsg6/s
The total number of ICMP messages which this interface attempted to send per second [ipv6IfIcmpOutMsgs].
iech6/s
The number of ICMP Echo (request) messages received by the interface per second [ipv6IfIcmpInEchos].
iechr6/s
The number of ICMP Echo Reply messages received by the interface per second [ipv6IfIcmpInEchoReplies].
oechr6/s
The number of ICMP Echo Reply messages sent by the interface per second [ipv6IfIcmpOutEchoReplies].
igmbq6/s
The number of ICMPv6 Group Membership Query messages received by the interface per second [ipv6IfIcmpInGroup-
MembQueries].
igmbr6/s
The number of ICMPv6 Group Membership Response messages received by the interface per second [ipv6IfIcmpInGroup-
MembResponses].
ogmbr6/s
The number of ICMPv6 Group Membership Response messages sent per second [ipv6IfIcmpOutGroupMembResponses].
igmbrd6/s
The number of ICMPv6 Group Membership Reduction messages received by the interface per second [ipv6IfIcmpInGroup-
MembReductions].
ogmbrd6/s
The number of ICMPv6 Group Membership Reduction messages sent per second [ipv6IfIcmpOutGroupMembReductions].
irtsol6/s
The number of ICMP Router Solicit messages received by the interface per second [ipv6IfIcmpInRouterSolicits].
ortsol6/s
onbsol6/s
The number of ICMP Neighbor Solicitation messages sent by the interface per second [ipv6IfIcmpOutNeighborSolicits].
inbad6/s
The number of ICMP Neighbor Advertisement messages received by the interface per second [ipv6IfIcmpInNeighb-
orAdvertisements].
onbad6/s
The number of ICMP Neighbor Advertisement messages sent by the interface per second [ipv6IfIcmpOutNeighborAdver-
tisements].
With the EICMP6 keyword, statistics about ICMPv6 error messages are reported. Note that ICMPv6 statistics depend on sadc option
"-S IPV6" to be collected. The following values are displayed (formal SNMP names between square brackets):
ierr6/s
The number of ICMP messages per second which the interface received but determined as having ICMP-specific
errors (bad ICMP checksums, bad length, etc.)
[ipv6IfIcmpInErrors]
idtunr6/s
The number of ICMP Destination Unreachable messages received by the interface per second [ipv6IfIcmpInDestUn-
reachs].
odtunr6/s
The number of ICMP Destination Unreachable messages sent by the interface per second [ipv6IfIcmpOutDestUnreachs].
itmex6/s
The number of ICMP Time Exceeded messages received by the interface per second [ipv6IfIcmpInTimeExcds].
otmex6/s
The number of ICMP Time Exceeded messages sent by the interface per second [ipv6IfIcmpOutTimeExcds].
iprmpb6/s
The number of ICMP Parameter Problem messages received by the interface per second [ipv6IfIcmpInParmProblems].
oprmpb6/s
The number of ICMP Parameter Problem messages sent by the interface per second [ipv6IfIcmpOutParmProblems].
iredir6/s
The number of Redirect messages received by the interface per second [ipv6IfIcmpInRedirects].
With the UDP6 keyword, statistics about UDPv6 network traffic are reported. Note that UDPv6 statistics depend on sadc option
"-S IPV6" to be collected. The following values are displayed (formal SNMP names between square brackets):
idgm6/s
The total number of UDP datagrams delivered per second to UDP users [udpInDatagrams].
odgm6/s
The total number of UDP datagrams sent per second from this entity [udpOutDatagrams].
noport6/s
The total number of received UDP datagrams per second for which there was no application at the destination port
[udpNoPorts].
idgmer6/s
The number of received UDP datagrams per second that could not be delivered for reasons other than the lack of
an application at the destination port [udpInErrors].
The ALL keyword is equivalent to specifying all the keywords above and therefore all the network activities are reported.
-o [ filename ]
Save the readings in the file in binary form. Each reading is in a separate record. The default value of the filename parameter
is the current daily data file, the /var/log/sysstat/sadd file. The -o option is exclusive of the -f option. All the data
available from the kernel are saved in the file (in fact, sar calls its data collector sadc with the option "-S ALL". See
sadc(8) manual page).
-P { cpu [,...] | ALL }
Report per-processor statistics for the specified processor or processors. Specifying the ALL keyword reports statistics for
each individual processor, and globally for all processors.
Note that processor 0 is the first processor.
-p Pretty-print device names. Use this option in conjunction with option -d. By default names are printed as dev m-n where m and
n are the major and minor numbers for the device. Use of this option displays the names of the devices as they (should) appear
in /dev. Name mappings are controlled by /etc/sysstat/sys-stat.ioconf.
-q Report queue length and load averages. The following values are
displayed:
runq-sz
Run queue length (number of tasks waiting for run time).
ldavg-15
System load average for the past 15 minutes.
-r Report memory utilization statistics. The following values are displayed:
kbmemfree
Amount of free memory available in kilobytes.
kbmemused
Amount of used memory in kilobytes. This does not take into account memory used by the kernel itself.
%memused
Percentage of used memory.
kbbuffers
Amount of memory used as buffers by the kernel in kilo-bytes.
kbcached
Amount of memory used to cache data by the kernel in kilobytes.
kbcommit
Amount of memory in kilobytes needed for current work-load. This is an estimate of how much RAM/swap is needed
to guarantee that there never is out of memory.
%commit
Percentage of memory needed for current workload in relation to the total amount of memory (RAM+swap). This num-
ber may be greater than 100% because the kernel usually overcommits memory.
-R Report memory statistics. The following values are displayed:
frmpg/s
Number of memory pages freed by the system per second. A negative value represents a number of pages allocated by
the system. Note that a page has a size of 4 kB or 8 kB according to the machine architecture.
bufpg/s
Number of additional memory pages used as buffers by the system per second. A negative value means fewer pages
used as buffers by the system.
campg/s
Number of additional memory pages cached by the system per second. A negative value means fewer pages in the
cache.
kbswpused
Amount of used swap space in kilobytes.
%swpused
Percentage of used swap space.
kbswpcad
Amount of cached swap memory in kilobytes. This is memory that once was swapped out, is swapped back in but
still also is in the swap area (if memory is needed it doesn't need to be swapped out again because it is
already in the swap area. This saves I/O).
%swpcad
Percentage of cached swap memory in relation to the amount of used swap space.
-t When reading data from a daily data file, indicate that sar should display the timestamps in the original locale time of the
data file creator. Without this option, the sar command displays the timestamps in the user's locale time.
-u [ ALL ]
Report CPU utilization. The ALL keyword indicates that all the CPU fields should be displayed. The report may show the follow-
ing fields:
%user
Percentage of CPU utilization that occurred while executing at the user level (application). Note that this field
includes time spent running virtual processors.
%usr
Percentage of CPU utilization that occurred while executing at the user level (application). Note that this field
does NOT include time spent running virtual processors.
%nice
Percentage of CPU utilization that occurred while executing at the user level with nice priority.
%system
Percentage of CPU utilization that occurred while executing at the system level (kernel). Note that this field
includes time spent servicing hardware and software interrupts.
%sys
Percentage of CPU utilization that occurred while executing at the system level (kernel). Note that this field
does NOT include time spent servicing hardware or software interrupts.
%soft
Percentage of time spent by the CPU or CPUs to service software interrupts.
%guest
Percentage of time spent by the CPU or CPUs to run a virtual processor.
%idle
Percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.
Note: On SMP machines a processor that does not have any activity at all (0.00 for every field) is a disabled (offline) pro-
cessor.
-v Report status of inode, file and other kernel tables. The following values are displayed:
dentunusd
Number of unused cache entries in the directory cache.
file-nr
Number of file handles used by the system.
inode-nr
Number of inode handlers used by the system.
pty-nr
Number of pseudo-terminals used by the system.
-V Print version number then exit.
-w Report task creation and system switching activity.
proc/s
Total number of tasks created per second.
cswch/s
Total number of context switches per second.
-W Report swapping statistics. The following values are displayed:
pswpin/s
Total number of swap pages the system brought in per second.
pswpout/s
Total number of swap pages the system brought out per second.
-y Report TTY device activity. The following values are displayed:
prtyerr/s
Number of parity errors per second for current serial line.
brk/s
Number of breaks per second for current serial line.
ovrun/s
Number of overrun errors per second for current serial line.
Note that with recent 2.6 kernels, these statistics can be retrieved only by root.
ENVIRONMENT
The sar command takes into account the following environment variables:
S_TIME_FORMAT
If this variable exists and its value is ISO then the current locale will be ignored when printing the date in the report
header. The sar command will use the ISO 8601 format (YYYY-MM-DD) instead.
S_TIME_DEF_TIME
If this variable exists and its value is UTC then sar will save its data in UTC time (data will still be displayed in local
time). sar will also use UTC time instead of local time to determine the current daily data file located in the
/var/log/sysstat directory. This variable may be useful for servers with users located across several timezones.
EXAMPLESsar -u 2 5 Report CPU utilization for each 2 seconds. 5 lines are displayed.
sar -I 14 -o int14.file 2 10 Report statistics on IRQ 14 for each 2 seconds. 10 lines are displayed. Data are stored in a file called int14.file.
sar -r -n DEV -f /var/log/sysstat/sa16 Display memory and network statistics saved in daily data file 'sa16'.
sar -A Display all the statistics saved in current daily data file.
BUGS /proc filesystem must be mounted for the sar command to work.
