Column	Mutex Contention
Source	/usr/bin/mpstat
Description	If you see high levels of mutex contention, you need to identify both the locks in contention and the component of the workload that is causing the contention like the system calls or a network protocol stack. You can then try to change the workload to avoid the problem. Mutex rules from mpstat output: For each CPU, if smtx is less than 500, or system time is less than 20 percent, there is no problem. If smtx is over 500, and system time is over 20 percent, then look at the ratio of user-to-system time. If user time is greater than system time it is only a warning (amber) level. If system time is higher, it is a problem (red) level. Read the Adrian Cockcroft article on mutex contention. Ok GREEN if(smtx_value<500 || sys <20) Some contention AMBER if(smtx > 500 && usr>=sys) Mutex contention RED if(smtx > 500 && sys>=usr)
Caveats

srtdate_mpstat: 20041027095232	SGT "mpstat 30 2"
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0   97   0  650   430  300  417   30   49  182    0    85   16   6   0  79
  1   92   0    2   186  105  734   63  161   90    0   356   13   5   0  81
  4   89   0  632   182  100  709   63  159   80    0   319   14   5   0  81
  5   89   0    5   184  100  728   65  168   92    0   333   13   5   0  81
  6   83   0  595   179  100  704   60  167   73    0   162   12   5   0  83
  7   83   0  808   180  101  723   61  174   83    0   179   12   5   0  83
  8   77   0  546   261  186  691   57  166   75    0    31   10   5   0  84
  9   74   0  762   420  343  687   57  174   86    0    32    9   6   0  85
 12   92   0  722   180  100  740   63  157   74    0   393   14   5   0  81
 13   92   0  128   185  103  741   64  166   95    0   404   13   5   0  81
 14   88   0  665   183  100  730   64  166   83    0   291   12   5   0  82
 15   86   0   62   190  105  746   65  172   98    0   282   12   5   0  83
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0  274   2 4146   494  300  861   98  155  179    0  2439   49  14   1  36
  1  296   3 1907   339  124 1205  164  282   76    0  3305   44  14   0  42
  4  227   3 3074   337  100 1292  180  307   80    0  2911   42   9   1  48
  5  217   3 1223   323  100 1288  172  291   75    0  3233   43  12   1  45
  6  220   3 2547   331  101 1335  177  308   71    0  2945   43   9   1  47
  7  242   2 2331   328  101 1252  175  289   71    0  3305   44  10   1  45
  8  180   3 2040   494  290 1239  155  293   84    0  3026   39  13   0  48
  9  212   3 2207  1029  811 1277  159  315   75    0  3082   35  13   1  51
 12  371   3 3256   315  101 1168  166  286   62    0  3460   43  14   1  42
 13  272   2 2282   327  103 1357  174  305   78    0  3370   42  13   1  44
 14  216   2 2037   341  101 1389  185  320   86    0  3248   39  13   0  47
 15  246   3 2219   359  124 1312  180  306   89    0  3274   39  13   0  47
enddate_mpstat: 20041027095232	SGT