[netperf-talk] CPU Utilization Issues

[Lentz, Benjamin A.]

Thank you for responding. Some responses below:

>>  
>> The CPU utilization is so high, in fact, that Oracle Clusterware
begins 
>> node eviction procedures when this test is performed while
Clusterware 
>> is running.

> How high is "so high?"

This is difficult to quantify but perfectly valid. The behavior we've
seen is that the system is unresponsive via ssh- even when the
interfaces being load tested are different from those being used to
access the system via SSH. It also causes a hang long enough to make
Oracle Clusterware believe that the node has hung and that it needs to
begin the node eviction process in order for the cluster's data
integrity to be maintained. 

> The netperf UDP_STREAM test has no flow-control of its own.  Netperf 
> will sit there and send UDP datagrams as fast as it can.  If there is
no 
> intra-stack flow control, it will take a core to 100% CPU utilization.

> If the path length of the stack is long enough relative to the "oomph"

> of the core, it will go to 100% utilization on a core even on a stack 
> with intra-stack flow control if that bottleneck is reached before the

> link itself.

Is it possible that our interconnect throughput is over running the
throughput in other areas of the system?

> Does Oracle Clusterware try to send traffic through the same NIC while

> the test is running?

Yes, indeed, and this is another reason why we believe there is a
conflict with testing the interconnect with netperf whilst Clusterware
is running. Oracle has confirmed that we shouldn't be doing this at the
same time.

However! Our vendor argues that they can indeed test with netperf while
Clusterware is running, and they identify this as a problem with our
environment.

> Out of curiousity, what led you to want to run a UDP_STREAM test over 
> say a TCP_STREAM test?

This is a good question. Our vendor is following Oracle Metalink
Documents 810394.1 and 563566.1, which appear to make reference to
netperf.

Oddly enough, all TCP_STREAM testing the vendor did seems to work fine. 

> Not sure what you mean by certify.  I have used netperf scores of
times 
> under RHEL5.3 x86_64, with a number of different NICs, but not with an

> IBM 3850.

>> We've tried asking Red Hat about this, and they've reported that they
do 
>> not support netperf. They suggested iperf.

> I'll have to have a talk with some of the RHEL folks about that... :)
I 
> do know there are at least a few folks at Red Hat who are quite fond
of 
> netperf.

Certainly! We as a Oracle and Red Hat customer, wishing to provide our
vendor with compelling evidence that our interconnect is actually
operating correctly, find ourselves entirely at the mercy of your
utility; our vendor refuses to make further progress with our systems
without being 100% certain that netperf confirms that our network
connection between our two systems is operating flawlessly.