[netperf-talk] How can I get a designated throughput by netperf

[Rick Jones]

On 07/22/2011 02:42 AM, Haifeng Li wrote:
> Hi Rick,
> *I will comment your questions and list my issues during my test today.*
> **
> *I executed following command to configure netperf, so I have enabled 
> "spin interval" in the test. *
> /*./configure --enable-histogram --enable-demo --enable-intervals 
> --enable-spin --enable-burst*/
> **

I see the "spin interval" in the headers now, confirming that it wasn't
before.  I wasn't really asking you to use it, only if you were using
it.  Spinning allows much finer grain on the interval time, but it does
so at the cost of considerable CPU utilization - rather than sleep
netperf spins in a loop calling gettimeofday() until the interval.  So,
unless you need sub-interval-timer granularity on the interval, don't
actually use --enable-spin.

> *I installed netperf client on one PC, and netperf server on other PC, 
> the client and server pass through a Gbps firewall, the PC system as below:*
> 1. OS: Red Hat Enterprise Linux 5
> 2. CPU: Intel(R) Core(TM)2 Duo CPU E7300 @ 2.66GHz
> 3. Memory: 2G
> 4: NIC: Gbps/full duplex

OK.  I forget, does RHEL5 have high-res timers (eg 1ms) or does it still
have 10ms granularity?

> *As you said, "/myself would probably go with much smaller, more 
> frequent bursts."/ I tested following command again, seems throughput 
> float percentage less than before, so you are right about this.*
> /[root at localhost ~]# netperf -t TCP_STREAM -H 10.3.0.1 -b 1 -w 10 -D 5 
> -l 60 -f m -c -C -- -m 1000k
> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.3.0.1 
> (10.3.0.1) port 0 AF_INET : histogram : spin interval : demo
> Interim result: 797.98 10^6bits/s over 5.00 seconds
> Interim result: 799.82 10^6bits/s over 5.00 seconds
> Interim result: 799.94 10^6bits/s over 5.00 seconds
> Interim result: 800.00 10^6bits/s over 5.00 seconds
> Interim result: 799.96 10^6bits/s over 5.00 seconds
> Interim result: 800.04 10^6bits/s over 5.01 seconds
> Interim result: 800.00 10^6bits/s over 5.00 seconds
> Interim result: 799.89 10^6bits/s over 5.00 seconds
> Interim result: 799.98 10^6bits/s over 5.00 seconds
> Interim result: 800.00 10^6bits/s over 5.01 seconds
> Interim result: 800.00 10^6bits/s over 5.00 seconds
> Recv Send Send Utilization Service Demand
> Socket Socket Message Elapsed Send Recv Send Recv
> Size Size Size Time Throughput local remote local remote
> bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB
> 
> 87380 16384 1000000 60.00 799.67 66.18 29.90 13.558 6.127

Good.

> /*As you said,/"Things that might be happening - TCP retransmissions 
> keeping the congestion window small, the congestion window resetting 
> after idle", /Yes, the throughput would be dropped off if we try to 
> burst too high throughput but it was failure, please see below real test 
> scenario.*
> //
> /try to burst 8000mbps on a Gbps network /
> /[root at localhost ~]# netperf -t TCP_STREAM -H 10.3.0.1 -b 100 -w 100 -D 
> 10 -l 120 -f m -c -C -- -m 1000k
> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.3.0.1 
> (10.3.0.1) port 0 AF_INET : histogram : spin interval : demo
> Interim result: 926.48 10^6bits/s over 10.01 seconds
> Interim result: 924.30 10^6bits/s over 10.02 seconds
> Interim result: 923.43 10^6bits/s over 10.01 seconds
> Interim result: 923.80 10^6bits/s over 10.00 seconds
> Interim result: 923.87 10^6bits/s over 10.00 seconds
> Interim result: 927.39 10^6bits/s over 10.01 seconds
> Interim result: 926.42 10^6bits/s over 10.01 seconds
> Interim result: 924.26 10^6bits/s over 10.02 seconds
> Interim result: 924.28 10^6bits/s over 10.01 seconds
> Interim result: 922.79 10^6bits/s over 10.01 seconds
> Interim result: 927.12 10^6bits/s over 10.00 seconds
> Recv Send Send Utilization Service Demand
> Socket Socket Message Elapsed Send Recv Send Recv
> Size Size Size Time Throughput local remote local remote
> bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB/
> //
> /87380 16384 1000000 120.03 924.55 13.71 22.40 2.429 3.969
> [root at localhost ~]# netperf -t TCP_STREAM -H 10.3.0.1 -b 100 -w 100 -D 
> 10 -l 120 -f m -c -C -- -m 1000k
> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.3.0.1 
> (10.3.0.1) port 0 AF_INET : histogram : spin interval : demo
> Interim result: 197.46 10^6bits/s over 11.14 seconds
> Interim result: 194.77 10^6bits/s over 10.15 seconds
> Interim result: 201.96 10^6bits/s over 10.06 seconds
> Interim result: 201.90 10^6bits/s over 10.02 seconds
> Interim result: 197.40 10^6bits/s over 10.21 seconds
> Interim result: 198.08 10^6bits/s over 10.06 seconds
> Interim result: 208.42 10^6bits/s over 10.02 seconds
> Interim result: 195.48 10^6bits/s over 10.68 seconds
> Interim result: 186.57 10^6bits/s over 10.46 seconds
> Interim result: 209.31 10^6bits/s over 10.05 seconds
> Interim result: 194.13 10^6bits/s over 10.71 seconds
> Recv Send Send Utilization Service Demand
> Socket Socket Message Elapsed Send Recv Send Recv
> Size Size Size Time Throughput local remote local remote
> bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB/
> //
> /87380 16384 1000000 120.04 199.12 5.95 11.63 4.897 9.567 /

> *But sometimes the throught still was down by correct command, like 
> below, we don't know why was it.*
> /[root at localhost ~]# netperf -t TCP_STREAM -H 10.3.0.1 -b 100 -w 1000 -D 
> 5 -l 60 -f m -c -C -- -m 1000k

Why do you call that the correct command?  I thought we have established
that large, infrequent bursts are not as good to use as smaller, more
frequent bursts?

And why do you keep changing the demo interval? Sometimes it is 10
seconds, sometimes 5.  And the same question for the run time - why do
you keep changing that?

Did you happen to run netstat before and after that test to see if there
were any retransmissions?

> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.3.0.1 
> (10.3.0.1) port 0 AF_INET : histogram : spin interval : demo
> Interim result: 107.06 10^6bits/s over 5.08 seconds
> Interim result: 104.08 10^6bits/s over 5.07 seconds
> Interim result: 103.97 10^6bits/s over 5.00 seconds
> Interim result: 103.40 10^6bits/s over 5.03 seconds
> Interim result: 101.34 10^6bits/s over 5.13 seconds
> Interim result: 106.22 10^6bits/s over 5.05 seconds
> Interim result: 103.13 10^6bits/s over 5.20 seconds
> Interim result: 104.17 10^6bits/s over 5.15 seconds
> Interim result: 103.54 10^6bits/s over 5.10 seconds
> Interim result: 104.06 10^6bits/s over 5.07 seconds
> Interim result: 103.42 10^6bits/s over 5.03 seconds
> Recv Send Send Utilization Service Demand
> Socket Socket Message Elapsed Send Recv Send Recv
> Size Size Size Time Throughput local remote local remote
> bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB/
> //



> *As you said, /"The autotuning may not be growing the window 
> sufficiently for you depending on the RTT of your network. You could try 
> a fixed socket buffer size large enough to hold each burst"./ I try to 
> send the socket buffer size by -s and -S option, but the troughtputs are 
> different between second commands, can other option/parameter/way avoid 
> this problem? BTW, the throughput of UDP are always same and stable 
> during the test.
> *
> /[root at localhost expect]# netperf -t TCP_STREAM -H 10.3.0.1 -b 1 -w 10 
> -D 5 -l 60 -f m -c -C -- -s 256k -S 256k -m 1000k
> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.3.0.1 
> (10.3.0.1) port 0 AF_INET : histogram : spin interval : demo
> Interim result: 796.24 10^6bits/s over 5.00 seconds
> Interim result: 799.99 10^6bits/s over 5.00 seconds
> Interim result: 799.79 10^6bits/s over 5.00 seconds
> Interim result: 799.98 10^6bits/s over 5.00 seconds
> Interim result: 799.99 10^6bits/s over 5.00 seconds
> Interim result: 799.96 10^6bits/s over 5.00 seconds
> Interim result: 799.99 10^6bits/s over 5.00 seconds
> Interim result: 800.00 10^6bits/s over 5.01 seconds
> Interim result: 799.99 10^6bits/s over 5.01 seconds
> Interim result: 800.00 10^6bits/s over 5.00 seconds
> Interim result: 800.00 10^6bits/s over 5.00 seconds
> Recv Send Send Utilization Service Demand
> Socket Socket Message Elapsed Send Recv Send Recv
> Size Size Size Time Throughput local remote local remote
> bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB/
> //
> /262142 262142 1000000 60.00 799.62 18.32 12.28 3.753 2.516
> [root at localhost expect]# netperf -t TCP_STREAM -H 10.3.0.1 -b 1 -w 10 -D 
> 5 -l 60 -f m -c -C -- -s 256k -S 256k -m 1000k
> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.3.0.1 
> (10.3.0.1) port 0 AF_INET : histogram : spin interval : demo
> Interim result: 95.18 10^6bits/s over 5.04 seconds
> Interim result: 96.11 10^6bits/s over 5.08 seconds
> Interim result: 96.53 10^6bits/s over 5.06 seconds
> Interim result: 95.98 10^6bits/s over 5.00 seconds
> Interim result: 96.09 10^6bits/s over 5.08 seconds
> Interim result: 95.79 10^6bits/s over 5.09 seconds
> Interim result: 96.11 10^6bits/s over 5.08 seconds
> Interim result: 96.54 10^6bits/s over 5.06 seconds
> Interim result: 96.49 10^6bits/s over 5.06 seconds
> Interim result: 95.85 10^6bits/s over 5.01 seconds
> Interim result: 96.21 10^6bits/s over 5.07 seconds
> Recv Send Send Utilization Service Demand
> Socket Socket Message Elapsed Send Recv Send Recv
> Size Size Size Time Throughput local remote local remote
> bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB/
> //
> /262142 262142 1000000 60.02 96.07 0.70 1.49 1.194 2.545 /

If one is going to make certain that the burst will fit in the socket
buffers, the socket buffers have to be at least as large as the burst.
So, if you have a burst of 1, and a send size of 1000k, the socket
buffer should be at least 1000k.

rick jones