[netperf-talk] CPU Utilization Issues

[Brandeburg, Jesse]

On Tue, 9 Feb 2010, Andrew Gallatin wrote:
> Rick Jones wrote:
> 
> > Anyhow, latency, especially small packet latency, is dominated by 
> > path-length vs CPU "oomph" and the interrupt coalescing decisions made 
> > by the NIC/driver.  An older writeup at:
> 
> One thing which some people may find surprising is that at least on
> recent Intel Nehalem based servers, I've seen roughly a factor of
> 2 improvement in UDP_RR numbers simply by disabling all powersaving
> in the BIOS (over and above disabling interrupt coalescing on the NIC).
> I'm guessing there are many microseconds of latency to wake up a
> slumbering core to handle a NIC interrupt and/or SMP IPI.

That and the other thing I would add is that on your 3850 inter-node (and 
to a lesser extent inter-core) communication adds lots of cpu and possibly 
latency.

Related to that is that particularly with kernels < 2.6.27, that do no 
support multiqueue transmit, if you have many threads all accessing a 
single transmit queue, there can be a lot of contention.  The best way to 
confirm this and to mitigate is to AFFINITIZE.

affinitize the memory allocations by the driver (f.e. e1000e)
taskset 1 modprobe e1000e
taskset 1 ifconfig eth0 up
taskset 2 ifconfig eth1 up
taskset 4 ifconfig eth2 up
taskset 8 ifconfig eth3 up

grep eth0 /proc/interrupts
for each of the interrupts in use (for e1000e, just one)
echo 1 > /proc/irq/<interrupt for eth0>/smp_affinity
echo 2 > /proc/irq/<interrupt for eth1>/smp_affinity
etc

double check that your udp mss is actually 1472 (somehow)

netperf -T0,0 -t UDP_RR -C -c -P0 -- -m 1472 

the -T0,0 forces netperf to run locally on cpu 0 and netserver to run 
remotely on cpu0

on 2.6.18 (EL5.3) you don't have the qdisc yield patch like is in 5.4, 
which can also make for near softlockups with many transmit threads.

do you have any iptables loaded?  are you using HPET timers?
lsmod |grep table
dmseg|grep time.c