[netperf-talk] cpu utilization on solaris

[Rick Jones]

Eric Davis wrote:
> Hi Rick,
> 
> ...
> I occasionally see cpu utilizations over 100% when running multiple
> netperf instances simultaneously.  For example here is some output from
> a tcp_rr w/ burst (note the formatting was performed by a script I wrote
> which is aggregate the output from each instance):
> 
>   ** TX UDP RR BURST 1b
>   ** netperf -t UDP_RR -H 172.16.0.20 -l 10 -C -c -- -S 1M -s 1M -b 32 -m 1
> 
>   Local /Remote
>   Socket Size   Request Resp.  Elapsed Trans.   CPU    CPU    S.dem   S.dem
>   Send   Recv   Size    Size   Time    Rate     local  remote local   remote
>   bytes  bytes  bytes   bytes  secs.   per sec  % M    % M    us/Tr   us/Tr
> 
>   1048576 1048576 1       1      9.99    21813.50   98.38  113.10 90.202  103.700
>   1048576 1048576
>   1048576 1048576 1       1      10.00   22229.81   98.38  113.11 88.514  101.764
>   1048576 1048576
>   1048576 1048576 1       1      9.99    18928.41   98.38  113.07 103.952  119.471
>   1048576 1048576
>   1048576 1048576 1       1      9.99    21581.20   98.38  113.07 91.173  104.783
>   1048576 1048576
>   1048576 1048576 1       1      9.99    17316.84   98.38  113.08 113.624  130.598
>   1048576 1048576
>   1048576 1048576 1       1      9.99    19919.87   98.38  112.90 98.775  113.352
>   1048576 1048576
>   1048576 1048576 1       1      10.00   18080.92   98.37  112.91 108.812  124.893
>   1048576 1048576
>   1048576 1048576 1       1      10.00   28672.47   96.32  105.48 67.185  73.578
>   1048576 1048576
> 
> As you can see I'm hammering the system.  Both sides are x86 running
> Solaris 10u7.  So, question is how is netperf calculating the CPU
> utilization to be over 100% on the remote system?

IIRC the "M" means that the src/netcpu_kstat10.c CPU utilization code is in use. 
There have been some squirrly things done there to workaround issues with 
certain times being measured overlapping with others, perhaps that is happening 
here.  Hopefully some folks with greater Solaris exposure will be able to 
comment.  There may be some sort of unfortunate interaction with PAD_TIME?

Enabling debug output on the netserver(s) might be helpful - launch netserver 
with -d (and netperf with -d too IIRC) and see what it is saying about its CPU 
utilization calculations.

happy benchmarking,

rick jones