Performance Monitoring and Tuning

mpstat

mpstat - report per-processor statistics 

 mpstat 1 2
 CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  6   93  57  617   414  302  452   14   73   25    0   284    2   3  46  48
  7   83  49  444   557  542  452   18   73   23    0   285    2   5  44  49

prstat
http://developers.sun.com/solaris/articles/prstat.html
The prstat -s cpu -n 3 command is used to list the three processes that are consuming the most CPU resources 

 $ prstat -s cpu -n 3
 PID	USERNAME	SIZE	RSS	STATE	PRI	NICE	TIME	CPU	PROCESS/NLWP
 13974	kincaid	888K	432K	run	40	0	36:14.51	67%	cpuhog/1
 27354	kincaid	2216K	1928K	run	31	0	314:48.51	27%	server/5
 14690	root	136M	46M	sleep	59	0	0:00.59	2.3%	Xsun/1

sar 

  -u for processor statistics
  -r for memory
  -k Kernel memory use

vmstat 

 # vmstat 1 4
  kthr      memory            page            disk          faults      cpu
  r b w   swap  free  re  mf pi po fr de sr f0 s0 s1 --   in   sy   cs us sy id
  0 0 0 2708524 711520 0   1  0  0  0  0  0  0  0  0  0  342  100  123  0  2 98
  0 0 0 2691220 693244 1  44  0  0  0  0  0  0  0  0  0  355  202  150  0  1 99
  0 0 0 2691220 693244 0   0  0  0  0  0  0  0  0  0  0  346  140  125  0  1 99
  0 0 0 2691220 693244 0   0  0  0  0  0  0  0  0  0  0  350  141  129  1  0 99

The fields for vmstat's display are 

 kthr   Number of kernel threads in each of the three following states
   r    run queue
   b    blocked kernel threads waiting for IO
   w    idle processes that have been swapped at some time

 Memory Report on usage of virtual and real Memory
   swap available swap space in KB
   free size of the free list in KB

 Page
   re  pages reclaimed from free list
   pi  pages paged in from filesystem or swap device
   po  pages paged out to filesystem or swap device
   fr  pages that have been freed or destroyed
   de  pages freed after writes

 CPU   Percentage usage of CPU
  us   user time
  sy   system time
  id   idle time

truss
The truss utility executes the specified command and produces a trace of the system calls it performs, the signals it receives, and the machine faults it incurs. Each line of the trace output reports either the fault or signal name or the system call name with its arguments and return value(s). 

 -o outfile  -- Send the output to outfile instead of standard out
 -e shows the enviroment strings that are passed on each system call
 -d Includes time stamps along with each line of trace out-put
 -c counts systems calls, faults and signals instead of displaying 
 -f to force truss to go into forked process 
 -p <pid> to attach to one or more process already running

To attach the truss to already running process 

 # truss -p <pid>

Memory and Paging spaces

To find out the total memory 

 # prtconf | grep Mem
 Memory size: 8192 Megabytes

sar -r gives report of Unused memory pages (multiply by the pagesize to get the actual value) and number of disk blocks (512bytes) for page swapping (freeswap) 

 # pagesize
 8192 

 # sar -r 1 4
 14:51:30 freemem freeswap
 14:51:31  537978 37760301
 14:51:32  537978 37760301
 14:51:33  538037 37760824
 14:51:34  538037 37760824

 Average   538007 37760563

Swap spaces

Swap space usage: With the exception of 'swap -l', most utilities on Solaris use 'swap' to mean 'Virtual memory', not the space on disk dedicated for backing store. So, swap -s, df -k and vmstat commands calculate swap space like below.
swap = physical memory + swap space - (minus ) memory used by OS/Kernel

To list all the swap spaces

  1. swap -l

swapfile dev swaplo blocks free
/dev/dsk/c0t1d0s4 32,12 16 4198368 4191696
/dev/dsk/c0t1d0s5 32,13 16 4198368 4191888
/dev/dsk/c0t1d0s1 32,9 16 4198368 4190192
/dev/dsk/c0t1d0s3 32,11 16 4198368 4192000

  1. swap -s

total: 244496k bytes allocated + 10904k reserved = 255400k used, 11254752k available