[netperf-talk] netperf port numbers

[Rick Jones]

On 08/25/2011 04:27 PM, Vishal Ahuja wrote:
> Hi Rick,
> Thanks for your input! I am still not clear why running TCP does not
> overwhelm the receiver (just a single core is enabled). As in, I don't
> see ksoftirqd running, nor are there are any packet loss in the ring
> buffer. Even though there are no losses, the throughput achieved by TCP
> is not great, and it does not increase even if I increase the number of
> flows (as shown below). The way I was thinking: Since netperf is the
> only network application running, and the RTT is negligible, the window
> size will grow quickly. Once the window size is large enough, the burst
> will cause packet loss (due to high interrupt load), and then the
> throughput would plummet. This will not happen in a 1 Gbps link, but it
> should in a 10 Gbps link. Where am I going wrong?

You underestimate the power of end-to-end flow control.  TCP has it.

Yes, under Linux, the receiving TCP will increase its advertised window 
- based on how quickly the receiving application is pulling data out of 
the socket.  If the application is not draining the socket, the window 
will not grow.

At the same time (again under Linux) the sending TCP is growing its 
SO_SNDBUF based on similar heuristics.

If you upgrade to version 2.5.0 of netperf (I'm assuming you are not 
there based on the output not including "MIGRATED") you will be able to 
see what the socket buffer sizes became during the test:

http://www.netperf.org/svn/netperf2/tags/netperf-2.5.0/doc/netperf.html#Omni-Output-Selection

$ src/netperf -H 192.168.1.3 -l 1 -- -k 
throughput,rsr_size_req,rsr_size,rsr_size_end,lss_size_req,lss_size,lss_size_end
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 
192.168.1.3 (192.168.1.3) port 0 AF_INET
THROUGHPUT=940.71
RSR_SIZE_REQ=-1
RSR_SIZE=87380
RSR_SIZE_END=2014880
LSS_SIZE_REQ=-1
LSS_SIZE=16384
LSS_SIZE_END=1571560


And, when the sending TCP fills the window, it will *stop*, waiting for 
a window update from the receiver, which will not happen any faster than 
the receiver can process data.

Thus, the TCP window will not grow any faster than the receiver can handle.

And there is also the presence of the congestion window - true it does 
not *prevent* packet losses, but it will minimize them.

Also, when you switch to -m 1470 with TCP a few things happen:

*) You likely diminish the effectiveness of TSO which will likely prefer 
larger send sizes
*) 1470 is a poor fit to the probable MSS of 1448 bytes (I'm assuming 
you've not done something silly like disable TCP timestamps)
*) depending on how bogus the implementation of the Nagle Algorithm is 
in Linux, you may run afoul of it and the interaction with delayed ACK 
heuristics  if adding a test-specific -D option noticeably increases 
throughput when the send size is > MSS then there is an issue with the 
nagle implementation.

happy benchmarking,

rick jones

> *./parallellaunch.py "netperf -H 192.168.0.8 -L 192.168.0.9 -t
> TCP_STREAM -l 30 -- -m 1470"
> TCP STREAM TEST from 192.168.0.9 (192.168.0.9) port 0 AF_INET to
> 192.168.0.8 (192.168.0.8) port 0 AF_INET : demo
> Recv   Send    Send
> Socket Socket  Message  Elapsed
> Size   Size    Size     Time     Throughput
> bytes  bytes   bytes    secs.    10^6bits/sec*
> * 87380  16384   1470    30.00    2442.46
> *
> *vishal at vishal-desktop:~/Desktop$*
> <mailto:vishal at vishal-desktop:~/Desktop$>*./parallellaunch.py "netperf
> -H 192.168.0.8 -L 192.168.0.9 -t TCP_STREAM -l 30 -- -m 1470" "netperf
> -H 192.168.0.8 -L 192.168.0.9 -t TCP_STREAM -l 30 -- -m 1470"
> TCP STREAM TEST from 192.168.0.9 (192.168.0.9) port 0 AF_INET to
> 192.168.0.8 (192.168.0.8) port 0 AF_INET : demo
> TCP STREAM TEST from 192.168.0.9 (192.168.0.9) port 0 AF_INET to
> 192.168.0.8 (192.168.0.8) port 0 AF_INET : demo
> Recv   Send    Send
> Socket Socket  Message  Elapsed
> Size   Size    Size     Time     Throughput
> bytes  bytes   bytes    secs.    10^6bits/sec*
> * 87380  16384   9000    30.00    1271.02
> Recv   Send    Send
> Socket Socket  Message  Elapsed
> Size   Size    Size     Time     Throughput
> bytes  bytes   bytes    secs.    10^6bits/sec*
> * 87380  16384   1470    30.00    1321.21
> *
> *vishal at vishal-desktop:~/Desktop$*
> <mailto:vishal at vishal-desktop:~/Desktop$>*./parallellaunch.py "netperf
> -H 192.168.0.8 -L 192.168.0.9 -t TCP_STREAM -l 30 -- -m 1470" "netperf
> -H 192.168.0.8 -L 192.168.0.9 -t TCP_STREAM -l 30 -- -m 1470" "netperf
> -H 192.168.0.8 -L 192.168.0.9 -t TCP_STREAM -l 30 -- -m 1470" "netperf
> -H 192.168.0.8 -L 192.168.0.9 -t TCP_STREAM -l 30 -- -m 1470"
> TCP STREAM TEST from 192.168.0.9 (192.168.0.9) port 0 AF_INET to
> 192.168.0.8 (192.168.0.8) port 0 AF_INET : demo
> TCP STREAM TEST from 192.168.0.9 (192.168.0.9) port 0 AF_INET to
> 192.168.0.8 (192.168.0.8) port 0 AF_INET : demo
> TCP STREAM TEST from 192.168.0.9 (192.168.0.9) port 0 AF_INET to
> 192.168.0.8 (192.168.0.8) port 0 AF_INET : demo
> TCP STREAM TEST from 192.168.0.9 (192.168.0.9) port 0 AF_INET to
> 192.168.0.8 (192.168.0.8) port 0 AF_INET : demo
> Recv   Send    Send
> Socket Socket  Message  Elapsed
> Size   Size    Size     Time     Throughput
> bytes  bytes   bytes    secs.    10^6bits/sec*
>
> * 87380  16384   1470    30.00     690.67
> Recv   Send    Send
> Socket Socket  Message  Elapsed
> Size   Size    Size     Time     Throughput
> bytes  bytes   bytes    secs.    10^6bits/sec*
>
> * 87380  16384   1470    30.01     677.85
> Recv   Send    Send
> Socket Socket  Message  Elapsed
> Size   Size    Size     Time     Throughput
> bytes  bytes   bytes    secs.    10^6bits/sec*
>
> * 87380  16384   1470    30.01     687.25
> Recv   Send    Send
> Socket Socket  Message  Elapsed
> Size   Size    Size     Time     Throughput
> bytes  bytes   bytes    secs.    10^6bits/sec*
>
> * 87380  16384   9000    30.01     659.33
> *
>
> Thank you,
> Vishal