[netperf-talk] TCP_SENDFILE

[Rick Jones]

Simon Leinen wrote:
> Checking on some systems here, it seems that Rick is right - Solaris 8
> doesn't have sendfile(), but Solaris 9 and later do.  You need to link
> with -lsendfile to use it (which doesn't exist on Solaris 8).
> 
> By the way, just today I stumbled across a Web page with recent
> NETPERF results on a Solaris 11 (snapshot) Dual Opteron machine using
> PCI-Express 10GE adapters... quite impressive
> 
> http://www.myri.com/Myri-10G/10gbe_performance.html

Not bad as netperf results writeups go.  Still missing a bit of data 
right there with the results, but better than many I've seen.

As far as TSO goes, my understanding is that 2.6.9 kernels aren't quite 
where one wants to be - that Linux had lower 1500 byte MTU TCP_STREAM 
numbers even with TSO suggests something wasn't completely happy.

The bit about Solaris not having TSO is "accurate" but perhaps a bit 
misleading - they have a similar, software-only feature called "Multi 
Data Transmit" whereby TCP can send a chain of segments to the driver at 
one time.  Upside is no need for explicit HW support, and no need for 
much "oomph" on the NIC.  Downside is it may not buy quite as much as 
full TSO, but that is hard to judge.

The "No CPU affinity on multi-cpu machines" bit in the Solaris 
description was a bit odd - I thought that reasonably contemporary 
netperfs knew how to do affinity under Solaris.  It also suggests that 
affinity was used under Linux, but was not stated explicitly.

I go to all that trouble to have some indication of what was used in 
those test banners and here people go not using it :)

For the 1500 byte tests, it _may_ have been better to bind the netperf 
(or netserver) to the CPU _other_ than the one taking the NIC interrupts 
to get a bit more parallelism.

Just a paranoid reminder - take the send side numbers of a UDP_STREAM 
test with a large dose of salt - different stacks do different things on 
outbound wrt intra-stack flow control and/or reporting ENOBUFS.  Best to 
stick with the receive side stats and consider using the 
--enable-intervals stuff if one needs to pace the sends.

rick jones