[netperf-talk] CPU utilization & tickless kernels?

[Andrew Gallatin]

On 12/07/12 16:54, Rick Jones wrote:
> On 12/07/2012 12:42 PM, Andrew Gallatin wrote:
>> I added some debugging code to
>> netcpu_procstat.c:calc_cpu_util_internal() to print out the
>> total ticks, and it is vastly different for each CPU !!!
>>
>> I then got to wondering:  /proc/stat has a summary line, which
>> appears to be what vmstat reads.  Why does netperf go through
>> the trouble to read all the CPUs individually?  Why can't you
>> just use the summary line?  Are there historical bugs with
>> the summary line?
> 
> The initial reasons are lost in the mists of time.  It may have been
> simply because the initial HP-UX version did per-CPU utilization.  Today
> one reason for the per-CPU tracking is to enable reporting the ID and
> utilization of the most utilized CPU on either side.  That is included
> for helping with things like a four CPU system being able to have 25%
> overall CPU utilization with either 25% across the board, two CPUs at
> 50% and two idle, or one at 100% and three idle, and all the other
> permuations.

I did not realize you could do this sort of thing with netperf2.


>> A very cheesy hack to use the summary line results in CPU
>> utilization which agrees with vmstat's summary.
>>
>> Drew
>>
>> PS: FWIW, mpstat also gives none-sense results on this system.
>> For example:
>>
>> 15:35:10     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal
>>   %guest   %idle
>> 15:35:11     all    0.00    0.00   60.61    0.00    0.00    7.27    0.00
>>     0.00   32.12
>> 15:35:11       0    0.00    0.00    0.00    0.00    0.00   18.75    0.00
>>     0.00   81.25
>> 15:35:11       1    0.99    0.00   99.01    0.00    0.00    0.00    0.00
>>     0.00    0.00
>>
>>
>> By the per-cpu counters, we've got (81.25 + 0) / 2 or 40.6% idle,
>> but the summary says 32% idle.
> 
> Well, misery loves company :)  As such, I'm glad that mpstat is
> similarly disagreeing.  Perhaps netperf needs to "interpret" the tick
> counts differently?  Scale them to the elapsed time somehow?  (Guessing)
>  Otherwise, perhaps the "bug" is in the source material.
> 
> Does top's per-CPU output look sane or mistaken?

No.  It looks like the same thing as mpstat.  The
summary does not match the per-cpu output.

Hmmm..

I played with trying to scale things based on the
max total_ticks I found, but I never got anything "reasonable"
in the case where one CPU is not pegged.  If one CPU is pegged,
then we get an accurate estimate of the max ticks/sec, and
I see CPU utilization similar to what I see for good old RHEL5 on
the same box...  (but ~10% higher than vmstat).

I'm pretty much convinced the kernel is lying to us.
I need to try a 3.7 kernel just to see if anything has
improved.

> rick
> 
> https://lkml.org/lkml/2012/2/6/20 has some issue with NOHZ and load
> averages - not sure if there is a relation there though.
> 
> I miss HP-UX's direct cycle counters, even if they were in an
> inconvenient lo/hi 32 bit format... "And you knew what you used then..."

Indeed, what a mess.

Drew