12.2 Overview of System Performance

12.3 Base Operation System Tools

The following commands are the focus throughout this section.

12.3.1 Using the vmstat Command

The vmstat command syntax is as follows:

vmstat [ -f ] [ -i ] [ -s ] [ PhysicalVolume ... ] [ Interval [ Count ] ]

The vmstat command reports statistics about kernel threads, virtual memory, disks, traps, and CPU activity. Reports generated by the vmstat command can be used to balance system load activity. These system-wide statistics (among all processors) are calculated either as averages for values expressed as percentages or as sums.

The PhysicalVolume parameter can be used to specify one to four names. Transfer statistics are given for each specified drive in the order specified. This count represents requests to the physical device. It does not imply an amount of data that was read or written. Several logical requests can be combined into one physical request.

If the vmstat command is invoked without flags, the report contains a summary of the virtual memory activity since system startup. If the -f flag is specified, the vmstat command reports the number of forks since system startup. The PhysicalVolume parameter specifies the name of the physical volume.

The Interval parameter specifies the amount of time in seconds between each report. The first report contains statistics for the time since system startup. Subsequent reports contain statistics collected during the interval since the previous report. If the Interval parameter is not specified, the vmstat command generates a single report and then exits. The Count parameter can only be specified with the Interval parameter. If the Count parameter is specified, its value determines the number of reports generated and the number of seconds apart. If the Interval parameter is specified without the Count parameter, reports are continuously generated. A Count parameter of zero (0) is not allowed.

The kernel maintains statistics for kernel threads, paging, and interrupt activity, which the vmstat command accesses. The disk input/output statistics are maintained by device drivers. For disks, the average transfer rate is determined by using the active time and number of transfers information. The percent active time is computed from the amount of time the drive is busy during the report.

Note

Both the -f and -s flags can be entered on the command line, but the system will only accept the first flag specified and will override the second flag.

Table 35 provides the key flags for the vmstat command.


Table 35: Key Flags for the vmstat Command

12.3.1.1 Examples

The following are some examples using the vmstat command.

  1. To display a summary of the statistics since boot, enter vmstat. A sample output follows:

    # vmstat
    kthr     memory             page              faults        cpu
    ----- ----------- ------------------------ ------------ -----------
     r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa
     0  0 19046  1554   0   0   0   0    0   0 117  310  30  0  1 99  1
    #
    


  2. To display five summaries at 2-second intervals, enter vmstat 2 5. A sample output follows:

    
    # vmstat 2 5
    kthr     memory             page              faults        cpu
    ----- ----------- ------------------------ ------------ -----------
     r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa
     0  0 19097  1498   0   0   0   0    0   0 117  310  30  0  1 99  1
     0  0 19097  1498   0   0   0   0    0   0 121  168  34  0  1 99  0
     0  0 19097  1498   0   0   0   0    0   0 126  118  34  0  0 99  0
     0  0 19097  1498   0   0   0   0    0   0 121  118  38  0  0 99  0
     0  0 19097  1498   0   0   0   0    0   0 122  121  34  0  0 99  0
    #
    
    


    The first summary (line one of the report) contains statistics for the time since boot.

  3. To display a summary of the statistics since boot, including statistics for logical disks hdisk0 and hdisk1, enter: vmstat hdisk0 hdisk1. A sample output follows:

    
    # vmstat hdisk0 hdisk1
    kthr     memory             page              faults        cpu      disk xfer
    ----- ----------- ------------------------ ------------ ----------- -----------
     r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa  1  2  3  4
     0  0 18461  3284   0   0   0   0    0   0 117  310  30  0  1 99  1  0  0
    #
    
    


  4. To display fork statistics, enter: vmstat -f. A sample output follows:
    # vmstat -f
    13887 forks
    #
    

  5. To display the count of various events, enter: vmstat -s. A sample output follows:

    
    # vmstat -s
        2205645 total address trans. faults
          46745 page ins
         135567 page outs
           7088 paging space page ins
          16737 paging space page outs
              0 total reclaims
         950333 zero filled pages faults
          12659 executable filled pages faults
         233034 pages examined by clock
             15 revolutions of the clock hand
          48272 pages freed by the clock
          27557 backtracks
              0 lock misses
             10 free frame waits
              0 extend XPT waits
          38657 pending I/O waits
         163907 start I/Os
         163907 iodones
       12734979 cpu context switches
       49535570 device interrupts
              0 software interrupts
              0 traps
      130379165 syscalls
    #
    
    


12.3.1.2 vmstat Report Output

Table 36 contains the column headings and their description for vmstat output.


Table 36: vmstat Output Parameters

Note

A large portion of real memory is utilized as a cache for file system data. It is not unusual for the size of the free list to remain small.

12.3.2 Using the iostat Command

The iostat command syntax is as follows:

iostat [ -d | -t ] [ PhysicalVolume ... ] [ Interval [ Count ] ]

The iostat command is used for monitoring system input/output device loading by observing the time the physical disks are active in relation to their average transfer rates. The iostat command generates reports that can be used to determine what changes should be made to the system configuration to better balance the input/output load between physical disks.

The first report generated by the iostat command provides statistics concerning the time since the system was booted.

Each subsequent report covers the time since the previous report. All statistics are reported each time the iostat command is run. The report consists of a TTY and CPU header row followed by a row of TTY and CPU statistics. On multiprocessor systems, CPU statistics are calculated system-wide as averages among all processors. A disks header row is displayed followed by a line of statistics for each disk that is configured. If the PhysicalVolume parameter is specified, only those names specified are displayed.

If the PhysicalVolume parameter is specified:

The first character in the PhysicalVolume parameter cannot be numeric.

The Interval parameter specifies the amount of time in seconds between each report. The first report contains statistics for the time since system startup (boot). Each subsequent report contains statistics collected during the interval since the previous report.

The Count parameter can be specified in conjunction with the Interval parameter. If the Count parameter is specified, the value of count determines the number of reports generated at Interval seconds apart. If the Interval parameter is specified without the Count parameter, the iostat command generates reports continuously.

The iostat command is useful in determining whether a physical volume is becoming a performance bottleneck and if there is potential to improve the situation. The % utilization field for the physical volumes indicates how evenly the file activity is spread across the drives. A high percentage utilization on a physical volume is a clear indication that there may be contention for this resource. Since the CPU utilization statistics are also available with the iostat report, the percentage of time the CPU is in I/O wait can be determined at the same time. Consider distributing data across drives if the I/O wait time is significant, and the disk utilization is not evenly distributed across volumes.

Note

Some system resource is consumed in maintaining disk I/O history for the iostat command. Use the sysconfig subroutine or SMIT to stop history accounting

Table 37 provides a list of common iostat command flags.


Table 37: Key Flags for the iostat Command

12.3.2.1 Examples

The following are examples of the iostat command usage.

  1. To display a single history since boot report for all tty, CPU, and Disks, enter:

    
    #iostat
    tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
              0.1         32.9               5.9     17.0       32.3      44.8
    
    Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
    hdisk1           3.7      34.5       1.9       4664       128
    hdisk0          46.5     526.3      40.2      68116      5048
    cd0              0.0       0.0       0.0          0         0
    #
    
    


  2. To display a continuous disk report at two second intervals for the disk with the name hdisk1, enter:
    iostat -d hdisk1 2
    

  3. To display six reports at two second intervals for the disk with the logical name hdisk1, enter:
    iostat -d hdisk1 2 6
    

  4. To display six reports at two second intervals for all disks, enter:
    iostat -d 2 6
    

  5. To display six reports at two second intervals for three disks named disk1, disk2, disk3, enter:
    iostat -d disk1 disk2 disk3 2 6
    

12.3.2.2 iostat Report Output

The iostat command generates two types of reports, the tty and CPU Utilization report and the Disk Utilization report. The meaning of the output parameters is shown in Table 38.


Table 38: iostat Output Parameters

For large system configurations where a large number of disks are configured, the system can be configured to avoid collecting physical disk input/output statistics when the iostat command is not executing. If the system is configured in the above manner, the first Disk report displays the message Disk History Since Boot Not Available instead of the disk statistics. Subsequent interval reports generated by the iostat command contain disk statistics collected during the report interval. Any TTY and CPU statistics after boot are unaffected. If a system management command is used to re-enable disk statistics, the first iostat command report displays activity from the interval starting at the point that disk input/output statistics were enabled.

12.3.3 Using the netstat Command

The netstat command syntax is as follows:

To display active sockets for each protocol or routing table information:

/bin/netstat [ -n ] [ { -A -a } | { -r -i -I Interface } ] [ -f AddressFamily ] [ -p Protocol ] [ Interval ] [ System ]

To display the contents of a network data structure:

/bin/netstat [ -m | -s | -u | -v ] [ -f AddressFamily ] [ -p Protocol ] [ Interval ] [ System ]

To display the packet counts throughout the communications subsystem:

/bin/netstat -D

The netstat command (Table 39) symbolically displays the contents of various network-related data structures for active connections. The Interval parameter, specified in seconds, continuously displays information regarding packet traffic on the configured network interfaces. The Interval parameter takes no flags. The System parameter specifies the memory used by the current kernel. Unless you are looking at a dump file, the System parameter should be /unix.


Table 39: Key Flags for the netstat Command

Note
  • In the statistics output, a N/A displayed in a field value indicates the count is not applicable. For the NFS/RPC statistics, the number of incoming packets that pass through RPC are the same packets that pass through NFS; so, these numbers are not summed in the NFS/RPC Total field, thus, the N/A. NFS has no outgoing packet or outgoing packet drop counters specific to NFS and RPC. Therefore, individual counts have a field value of N/A, and the cumulative count is stored in the NFS/RPC Total field.

  • The collision count for Ethernet interfaces is not supported.
  • 12.3.3.1 Examples

    The following are examples of netstat command usage.

    To display the routing table, use the command:

    
    # netstat -r
    Routing tables
    Destination      Gateway           Flags   Refs     Use  If   PMTU  Exp  Groups
    
    Route Tree for Protocol Family 2 (Internet):
    default          itsorusi.itsc.aus UG       14      706  tr0     -   -
    9.3/16           sv1051c.itsc.aust U         0        5  en0     -   -
    9.3.1/24         sv1051c.itsc.aust U        40     2616  tr0     -   -
    127/8            localhost         U         3      180  lo0     -   -
    192.168.1/24     sv1166f.itsc.aust UG        0        0  tr0     -   -
    
    Route Tree for Protocol Family 24 (Internet v6):
    ::1              ::1               UH        0        0  lo0 16896   -
    #
    
    

    To display the routing statistics use the command:

    
    # netstat -r -s
    routing:
            0 bad routing redirect
            0 dynamically created route
            0 new gateway due to redirects
            0 destination found unreachable
            0 use of a wildcard route
    #
    
    

    12.3.3.2 netstat Output Report

    The default display for active sockets shows the following items:

    Internet address formats are of the form host.port or network.port if a socket's address specifies a network but no specific host address. The host address is displayed symbolically if the address can be resolved to a symbolic host name while network addresses are displayed symbolically according to the /etc/networks file.

    NS addresses are 12-byte quantities consisting of a 4-byte network number, a 6-byte host number, and a 2-byte port number, all of which are stored in network standard format. For VAX architecture, these are word and byte reversed; for the Sun systems, they are not reversed.

    If a symbolic name for a host is not known, or if the -n flag is used, the address is printed numerically according to the address family. Unspecified addresses and ports appear as an * (asterisk).

    Interface Display (netstat -i)

    The interface display format provides a table of cumulative statistics for the following items:

    Routing Table Display (netstat -r)

    The routing table display format indicates the available routes and their statuses. Each route consists of a destination host or network and a gateway to use in forwarding packets. The routing table contains the following ten fields:

    Flags

    The flags field of the routing table shows the state of the route:

    U
    Up.
    H
    The route is to a host rather than to a network.
    G
    The route is to a gateway.
    D
    The route was created dynamically by a redirect.
    M
    The route has been modified by a redirect.
    L
    The link-level address is present in the route entry.
    c
    Access to this route creates a cloned route. This field only applies to AIX Version 4.2.1 or later.
    W
    The route is a cloned route. This field only applies to AIX Version 4.2.1 or later

    Direct routes are created for each interface attached to the local host.

    Gateway
    The gateway field for these entries shows the address of the outgoing interface.
    Refs
    Gives the current number of active uses for the route. Connection-oriented protocols hold onto a single route for the duration of a connection, while connectionless protocols obtain a route while sending to the same destination.
    Use
    Provides a count of the number of packets sent using that route.
    PMTU
    Gives the Path Maximum Transfer Unit (PMTU). This field only applies to AIX Version 4.2.1 or later.
    Interface
    Indicates the network interfaces utilized for the route.
    Exp
    Displays the time (in minutes) remaining before the route expires. This field only applies to AIX Version 4.2.1 or later.
    Groups
    Provides a list of group IDs associated with that route. This field only applies to AIX Version 4.2.1 or later.
    Netmasks
    Lists the netmasks applied on the system.

    12.4 Performance Analysis