[netperf-talk] CPU Utilization Issues

[Rick Jones]

Lentz, Benjamin A. wrote:
>>If you add a -c and -C option to netperf, what CPU utilization does it
> 
> 
>>report?
> 
> 
> I have contacted the folks necessary in order to bring down the
> clustering services again for another test... We'll see. Do we need to
> specify a [rate] value, or just leave it empty?

The CPU utilization measurements do not require a calibration phase 
under Linux so you can just leave it empty.

> 
> 
>>BTW, is this the system on which netperf is running that is thought to
>>be dead, or the one that is the target of the netperf test (ie where 
>>netserver is running)?
> 
> 
> This is a tricky question. I believe the Oracle node eviction process is
> operating under several mechanisms to determine node death... And we've
> seen all combinations in our testing. One of the other problems we're
> facing is that in a two-node cluster, determining which node is actually
> malfunctioning is difficult for the system... Adding a third node (or
> using odd-numbered quantity of nodes) solves this problem... We do not
> have the hardware for this at this time though.

One way to think about a netperf UDP_STREAM test without 
--enable-intervals configure-ation is a Denial of Service attack :)

>>Perhaps.  Can you provide information such as the output of 
>>/proc/interrupts and then /proc/irq/<irq>/smp_affinity for some of the
>> NICs?
> 
> Sure, here's one of our systems.
> 
> $ cat /proc/interrupts | grep eth
>  59:        103     326966    1241556    1240838         PCI-MSI  eth3
>  67:         47        302        458    3928039         PCI-MSI  eth5
> 146:        206          0    3655016       4794         PCI-MSI  eth0
> 178:        143    2179700     405245     224575         PCI-MSI  eth1
> 210:        103     296887    1163469    1343220         PCI-MSI  eth2
> 218:         34     655377       2258        767         PCI-MSI  eth4
> 
> $ for irq in 59 67 146 178 210 218; do sudo cat
> /proc/irq/$irq/smp_affinity; done
> 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000008
> 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000008
> 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000004
> 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000002
> 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000004
> 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000002
> 
> 
>>Is the irqbalance service running.
> 
> 
> Yes, it is.

Yep - it has been moving the interrupts around - the smp_affinity values 
are bitmasks.  I guess that the irqbalanced has not wanted to put them 
onto  CPU0 (notice no value of 0x1 in the masks) just CPUs 2, 3 and 4.

> 
>> You are 100% certain they are running a UDP_STREAM and not a 
>> TCP_STREAM test?  Are you using the exact same command line as they
>> are?  They haven't done something like ./configure
>> --enable-intervals and "paced" the UDP_STREAM test?
> 
> 
> Well, I have to believe what I am told, but yeah, the best information I
> have available is that we're running the exact same command between the
> two systems (one crashing Oracle, the other not). I have no clue about
> the ./configure options used, but I've posed the question to them via
> email.
> 
> 
> 
>>Out of curiousity, what led you to want to run a UDP_STREAM test over 
>>say a TCP_STREAM test?
> 
> 
> FWIW, all the Oracle inter-node communication is done over UDP inside
> the Oracle configuration.

Unless there is RDS, yes, that is my understanding.  Still, Oracle's 
communications are, IIRC not streams of 1472 byte messages.  They are 
request/response with upwards of 8KB in one direction. (That is vague 
recollection, not 100% certainty)  That being the case, some variation 
on the UDP_RR test is probably "better" than a UDP_STREAM test.  If 
Oracle's documentation is suggesting a UDP_STREAM test by all means 
suggest to them they need to get in touch with "Mr. Netperf" for some 
reeducation...


>>This is a good question. Our vendor is following Oracle Metalink
>>Documents 810394.1 and 563566.1, which appear to make reference to
>>netperf.
>>
>> Oddly enough, all TCP_STREAM testing the vendor did seems to work
>> fine. 
> 
> TCP_STREAM benefits from TCP's flow control.  It will not run wild and 
> potentially fill the drivers transmit queue.  Any other traffic will 
> then have a much better chance of actually getting through.
> 
> Versus a UDP_STREAM test, where other, non-netperf traffic (such as
> those used by Oracle to run OCFS2 or RAC clustering) will have a
> *difficult* time getting through? This would explain why TCP can
> co-exist and why UDP can not.

Indeed.  TCP tries to be "fair" - that is the whole impetus behind the 
congestion control and avoidance heuristics in TCP.

>> I love to see people using netperf.  However I have to remind folks
>> that netperf is a *performance benchmark* - it is *not* a
>> functional test tool.  Even though it can often uncover functional
>> problems in a stack or NIC or whatnot.
>>
>> If the desire is to see that the interconnect is working properly,
>> I would think that "successful" netperf tests without the
>> clustering software running would be sufficient.  Of course I am a
>> networking guy, not a clustering or database type :)
> 
> 
> Good answer :-)
> 
> Thanks for all your help thus far. It's greatly appreciated. 

You're welcome.

rick jones