[netperf-talk] Some question about TCP_RR test

[Rick Jones]

On 07/21/2011 03:20 AM, Hangbin Liu wrote:
> Hi all,
>
>     I got a network traffic flood in our lab yesterday when I use TCP_RR.
> Could you help me answer the questions and find the flood reason?

Traffic flood? With netperf TCP_RR?  FWIW, nothing that follows in the 
email suggests (at least on the surface) a netperf-induced traffic flood 
on your network.

> The cmd is:
> "netperf -6 -l 20 -H 2620:52:0:102f:207:43ff:fe10:760 -t TCP_RR -i 10,3
> -I 99,5 -r {1,16,32...},1 -s 0 -S 0".
> We test the TCP_RR option and set request package from 1 to 4381. And we
> often got a warning like:
> !!! WARNING
> !!! Desired confidence was not achieved within the specified iterations.
> !!! This implies that there was variability in the test environment that
> !!! must be investigated before going further.
> !!! Confidence intervals: Throughput      : 25.119%
> !!!                       Local CPU util  : 0.000%
> !!!                       Remote CPU util : 0.000%
>
> Local /Remote
> Socket Size   Request  Resp.   Elapsed  Trans.
> Send   Recv   Size     Size    Time     Rate
> bytes  Bytes  bytes    bytes   secs.    per sec
>
> 2048   256    1        1       20.00    6014.78
> 2048   256
>
> +++++++++++++++++++++++++++++++ Questions +++++++++++++++++++++++++++++++++
>
> 1. What's this warning means, does that have any influence for our test output?

The -I 99,5 was requesting that netperf attempt to be 99% 
certain/confident  that the average it was going to report was within 
+/- 2.5% of the "real" average for, in this case, transactions per 
second.  The -i 10,3 said that netperf should run at least three 
iterations of the test, and no more than 10, when attempting to acheive 
that level of confidence.

http://www.netperf.org/svn/netperf2/trunk/doc/netperf.html#index-g_t_002dI_002c-Global-27

> 2. And what the throughput means? I can't find it from man page and sometimes I
>   even got a :
> Confidence intervals: Throughput      : 654.493%
>
> why it can exceed 100%, is that have any problems for our lab?

Throughput is a generic term in that output - the same code is used to 
emit the warning for all tests.  In the case of the TCP_RR test the 
"throughput" is usually reported in transactions per second.

http://www.netperf.org/svn/netperf2/trunk/doc/netperf.html#Using-Netperf-to-Measure-Request_002fResponse

I slept too often in my Probability and Statistics class in college, but 
it can exceed 100% if the results are particularly variable.  The width 
of the confidence interval is expressed as a percentage of the average 
(the mean).  Now, this shows that my simple "divide the width by two and 
say +/-" is simplistic for it would be quite difficult indeed to have 
something that was -377ish percent of the average but still... :)

> 3. What's Trans Rate per sec number means? bits/s, Bits/s, Kb, Mb ???

A transaction here is the exchange of one request and one response.  So, 
it is just what it says - transactions per second.

> 4. How client performance affect throughput

How long is a piece of string?

There are a plethora of things happening in/with the client that can 
affect the performance of the netperf TCP_RR test.  They include, but 
are not limited to, and in no particular order:

*) the path length of the TCP/IP/driver stack
*) the DMA latency of the system
*) the performance of the processor
*) the width of the I/O bus to which the NIC is connected
*) the underlying HW/FW performance of the NIC
*) the presence or lack of offload features in the NIC
*) etc etc etc


> ++++++++++++++++++++++++++++++++++++++ Problem +++++++++++++++++++++++++++++++++
>
> Normally we get the warning from package sizes 256, but this time we got the
> warning from 1. Can you help me find the reason? Thanks!
>
> *Normal test:
>
> Socket Size   Request  Resp.   Elapsed  Trans.
> Send   Recv   Size     Size    Time     Rate
> bytes  Bytes  bytes    bytes   secs.    per sec
>
> 2048   256    1        1       20.00    8518.70
> 2048   256
> 2048   256    16       1       20.00    8269.85
> 2048   256
> 2048   256    32       1       20.00    8141.42
> 2048   256
> 2048   256    64       1       20.00    7905.91
> 2048   256
> !!! WARNING
> !!! Desired confidence was not achieved within the specified iterations.
> !!! This implies that there was variability in the test environment that
> !!! must be investigated before going further.
> !!! Confidence intervals: Throughput      : 10.275%
> !!!                       Local CPU util  : 0.000%
> !!!                       Remote CPU util : 0.000%
>
> 2048   256    128      1       20.00    6650.96
>
> *Test last night:
>
> !!! WARNING
> !!! Desired confidence was not achieved within the specified iterations.
> !!! This implies that there was variability in the test environment that
> !!! must be investigated before going further.
> !!! Confidence intervals: Throughput      : 25.119%
> !!!                       Local CPU util  : 0.000%
> !!!                       Remote CPU util : 0.000%
>
> Local /Remote
> Socket Size   Request  Resp.   Elapsed  Trans.
> Send   Recv   Size     Size    Time     Rate
> bytes  Bytes  bytes    bytes   secs.    per sec
>
> 2048   256    1        1       20.00    6014.78
> 2048   256
> !!! WARNING
> !!! Desired confidence was not achieved within the specified iterations.
> !!! This implies that there was variability in the test environment that
> !!! must be investigated before going further.
> !!! Confidence intervals: Throughput      : 12.254%
> !!!                       Local CPU util  : 0.000%
> !!!                       Remote CPU util : 0.000%
>
> 2048   256    16       1       20.00    6126.77
> 2048   256

It simply means that netperf, given the conditions of the test and the 
constraints of the command line, was unable to be "confident" that the 
actual average transaction rate - what one would see if one ran an 
infinite number of iterations was within the desired distance of what 
netperf actually measured.

It means there was more variability in the result (transaction rate) 
than the command line told netperf you wanted.  So, it issues the warning.

You can try increasing the number of iterations - the maximum is 30 (-i 
30,3).  You can try increasing the length of each iteration (-l 60). 
You might even try both.

There may also be some interesting interactions with the rather small 
socket buffer sizes the netperf command line is requesting - 2048 and 
256 are (IIRC) linux minimums, stemming from the "-s 0 and -S 0" 
test-specific options.  If the intent is to get the system default, then 
those options should be omitted, or given a value of -1.  Under Windows 
a value of "0" means "try to use copy avoidance" so many years ago that 
change was introduced into netperf.