[netperf-talk] ENOBUFS on linux?

[Rick Jones]

> It seems obvious that the kernel is dropping the packets due to them
> exceeding a queue size someplace.  In a BSD stack, sendto() would return
> ENOBUFS and netperf would report correct results.    I don't think this is a
> driver issue, as it happens with 3 different ethernet drivers.
> 
> Does anybody know how to get linux to throw an error in this circumstance?
> Based on some googling, I tried setting IP_RECVERR on the socket, but with no
> luck.

I'm guessing IP_RECVERR is to get ICMP messages.

As you might imagine, netperf relies entirely on either there being intra-stack 
flow-control or an error return on the sendto() calls for the UDP_STREAM stuff. 
    HP-UX 10.20 and before, being BSD-based, would emit an error return when the 
driver's transmit queue filled. HP-UX 11.0 and later does not, nor does it have 
intra-stack flow control.

It is clear that at least in some cases, Linux does have intra-stack 
flow-control - whether it is a bug that one does not get it with the larger 
socket buffer sizes I don't know, but suspect the fine folks at the netdev 
mailing list might.  Likely as not they would want to see something showing the 
issue on a 2.6.3X kernel.

It would seem that 64K for the socket buffer size request is OK - I wonder if 
65537 bytes remains OK - perhaps there is a 16 bit boundary crossing thing 
involved here.

happy benchmarking,

rick jones

> 
> (FWIW, this is RHEL 5.5 2.6.18-164).

Is that 5.5 or 5.4?  Anyway, on a 2.6.18-164-el5 kernel over a 1GbE interface 
being driven by bnx2 I do not see the "greater than linkrate" with a 96K send 
socket buffer, so there may be a driver component in there somewhere.   Still, 
ifconfig shows the txqueuelen as 1000 in this case and 96K of 1472 byte sends 
would be only 67 packets.  Perhaps being < txquelen is a coincidence.  If I go 
to -s 1M I get just a little bit of >> linkrate.  If I go to -s 2M I get 10X 
link-rate.  Of course, even 1MB as 1472 byte sends is < 1000, and even in the 
10X faster case I see no TX overruns reported, and the ethtool -S stats reported 
appear clean.

(I've set net.core.[wr]mem_max to 4194304 on this system)