David Krider Measuring System Performance

July 2, 2013 by · Leave a Comment 

There is a tool on open systems called sar. (On Linux, you can install the “sysstat” package to get access to it, and a lot of other utility programs in the same vein.) It does a lot of work in a small package. Just knowing everything it can do is intimidating. Understanding its output can be downright daunting. Luckily, Adrian Cockcroft — distinguished scientist of the old Sun corporation and current CTO of Netflix — wrote a book about the subject many years ago: Sun Performance and Tuning: Sparc & Solaris. Of course, it was specific to Solaris, but the general principles of identifying CPU, memory, and disk bottlenecks is still applicable to any general-purpose computer today.

First, `sar’ is a standalone tool that you can run to see just about any performance metric related to computers. Running the main commands of either `sar’ or `iostat’ can be done by giving the number of seconds to poll, and the number of times to run. For instance:

dkrider@workstation:~$ sar 1 5
Linux 3.8.0-25-generic (workstation)     06/19/2013     _x86_64_    (2 CPU)
03:41:30 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
03:41:31 PM     all      2.50      0.00      0.50      0.00      0.00     97.00
03:41:32 PM     all      3.52      0.00      0.00      2.01      0.00     94.47
03:41:33 PM     all      2.01      0.00      1.01      0.00      0.00     96.98
03:41:34 PM     all      2.02      0.00      0.51      0.00      0.00     97.47
03:41:35 PM     all      2.50      0.00      0.50      1.00      0.00     96.00
Average:        all      2.51      0.00      0.50      0.60      0.00     96.39

An important thing to realize is that the information given by the first line is junk. It takes two readings for the polling to settle down and start to make sense. Ignore it.

Second, you can enable the intermittent polling and saving of performance data in the background. On Ubuntu, for instance, you can edit /etc/default/sysstat, and change the “ENABLED” line to “true”, which will cause another utility called `sadc’ to poll the machine every 10 minutes and save the stats to /var/log/sysstat. Then, running `sar’ without any flags produces a listing of the basic metrics for the last day:

dkrider@cloud3:~$ sar
Linux 3.2.0-40-generic (cloud3)     06/19/2013     _x86_64_    (8 CPU)
12:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:05:01 AM     all      2.24      0.00      1.24     11.11      0.00     85.42
12:15:01 AM     all      2.60      0.00      1.42     11.57      0.00     84.41
12:25:01 AM     all      2.78      0.00      1.56     12.20      0.00     83.46
                        -------------- <SNIP> ---------------
01:45:01 PM     all      0.98      0.00      0.55      8.67      0.00     89.81
01:55:01 PM     all      0.99      0.00      0.56      9.24      0.00     89.21
02:05:01 PM     all      1.02      0.00      0.57      8.76      0.00     89.64
Average:        all      0.72      0.00      0.44     11.37      0.00     87.47

`iostat’ operates the same way:

dkrider@workstation:sysstat$ iostat 5 5
Linux 3.8.0-25-generic (workstation) 	06/25/2013 	_x86_64_	(2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           7.62    0.05    0.84    0.29    0.00   91.21

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               0.69         3.80         4.93    3584444    4657308
sdb               0.57         1.11         4.93    1043859    4657316
sdd               1.07         0.82        25.68     772207   24241929
sdc               1.23         5.20        25.68    4912319   24241929
md1               1.93         6.02        25.52    5682704   24090641
md0               0.90         4.87         4.73    4594593    4460504
dm-0              0.02         0.02         0.06      14628      56240
dm-1              0.02         0.02         0.06      14704      56248

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.90    0.00    0.40    0.30    0.00   98.40

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               0.40         2.40         0.00         12          0
sdb               0.00         0.00         0.00          0          0
sdd               0.80         0.00         6.80          0         34
sdc               0.80         0.00         6.80          0         34
md1               1.60         0.00         6.40          0         32
md0               0.00         0.00         0.00          0          0
dm-0              0.00         0.00         0.00          0          0
dm-1              0.00         0.00         0.00          0          0

(See what I mean about disregarding the first set of numbers?)

There’s another tool you may want to install, called `isag‘. It allows you to graph the accumulated logs of stats in /var/log/sysstat. I mention it first, because it’s probably in your distribution’s package manager. There’s another nice tool called kSar, which is a Qt-based graphing utility.

Speak Your Mind

Tell us what you're thinking...
and oh, if you want a pic to show with your comment, go get a gravatar!