[netperf-talk] CPU utilization & tickless kernels?

[Andrew Gallatin]

Hi Rick,

I'm running netserver on a crappy AMD E-350 (which is great for 10GbE
perf testing, since any small change results in a measurable difference
in bandwidth or CPU utilization).  The problem I'm seeing is that under
Ubuntu 12.10, net{perf,server} reports vastly different CPU utilization
from vmstat.

netperf (to AMD E-350) is reporting ~51%:

netperf260 -Hbam-my -tTCP_SENDFILE -F /boot/vmlinuz-3.7.0-rc7+ -T3,1 -C -c
TCP SENDFILE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to bam-my ()
port 0 AF_INET : cpu bind
Recv   Send    Send                          Utilization       Service
Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local
remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB   us/KB

 87380  65536  65536    10.02      8803.27   15.88    50.76    0.591
0.945


But vmstat is reporting ~65%

vmstat:
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy
id wa
<...>
 1  0      0 7308980  35188 258476    0    0     0     0 13611   16  1
65 35  0
 1  0      0 7308788  35188 258476    0    0     0     0 13533   16  0
64 36  0
 1  0      0 7308884  35188 258476    0    0     0     0 13656   12  1
63 35  0
 1  0      0 7308884  35188 258476    0    0     0     0 13646   20  1
64 35  0
 1  0      0 7308948  35188 258476    0    0     0     0 13661   10  1
64 35  0
 1  0      0 7308656  35188 258476    0    0     0    32 13662   29  1
64 34  1
 1  0      0 7308788  35188 258476    0    0     0     0 13661   12  1
64 36  0
 1  0      0 7308756  35188 258476    0    0     0     0 13623   24  0
65 35  0
 1  0      0 7308936  35188 258476    0    0     0     0 13599   78  1
64 35  0
 0  0      0 7331576  35188 258476    0    0     0     0 12831   45  0
61 39  0
<...>

I added some debugging code to
netcpu_procstat.c:calc_cpu_util_internal() to print out the
total ticks, and it is vastly different for each CPU !!!

I then got to wondering:  /proc/stat has a summary line, which
appears to be what vmstat reads.  Why does netperf go through
the trouble to read all the CPUs individually?  Why can't you
just use the summary line?  Are there historical bugs with
the summary line?

A very cheesy hack to use the summary line results in CPU
utilization which agrees with vmstat's summary.

Drew

PS: FWIW, mpstat also gives none-sense results on this system.
For example:

15:35:10     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal
 %guest   %idle
15:35:11     all    0.00    0.00   60.61    0.00    0.00    7.27    0.00
   0.00   32.12
15:35:11       0    0.00    0.00    0.00    0.00    0.00   18.75    0.00
   0.00   81.25
15:35:11       1    0.99    0.00   99.01    0.00    0.00    0.00    0.00
   0.00    0.00


By the per-cpu counters, we've got (81.25 + 0) / 2 or 40.6% idle,
but the summary says 32% idle.