[netperf-talk] global question concerning Netperf test and SMP support

[Rick Jones]

On 04/19/2012 04:37 AM, Simon Duboue wrote:
>  >On 04/04/2012 03:15 AM, Simon Duboue wrote:
>  >>
>  >> Could the total throughput be considered as the sum of the two obtained
>  >> results?
>  >
>  >Maybe :) There is no explicit synchronization in netperf. There are a
>  >few ways to address issues of skew error:
>  >
>  >http://www.netperf.org/svn/netperf2/trunk/doc/netperf.html#Using-Netperf-to-Measure-Aggregate-Performance
>  >
>  >The one I currently use the most is the "demo mode with post-processing"
>  >path, though that has an issue with UDP_STREAM - the interim results
>  >reported by netperf in a UDP_STREAM test will be sender side, not
>  >receiver side. In that case the (ab)use of confidence intervals path
>  >may be better (though take longer).
>  >
>  >While the local CPU util in your pair of UDP_STREAM tests was the same,
>  >the remote CPU util differed enough that I would not trust the results
>  >to have low skew error.
>
> Now I get the same version of netperf on both sides. The results seem to
> be in the same range as previously.
> I come back on what you say before concerning the use of the -i option.
> As expected, I have a warning for the confidence intervals not reached.
>
> netperf -i 10 -cC -p 17170 -l 30 -P 0 -H $1 -t TCP_STREAM -- -C &
> netperf -i 10 -cC -p 17170 -l 30 -P 0 -H $1 -t TCP_STREAM -- -C &
> netperf -i 10 -cC -p 17170 -l 30 -P 0 -H $1 -t TCP_STREAM -- -C &
> netperf -i 10 -cC -p 17170 -l 30 -P 0 -H $1 -t TCP_STREAM -- -C
>
> !!! WARNING
> !!! Desired confidence was not achieved within the specified iterations.
> !!! This implies that there was variability in the test environment that
> !!! must be investigated before going further.
> !!! Confidence intervals: Throughput : 11.9%
> !!! Local CPU util : 10.9%
> !!! Remote CPU util : 70.5%
>
> 87380 16384 16384 30.06 641.87 6.77 10.06 1.732 10.306
> !!! WARNING
> !!! Desired confidence was not achieved within the specified iterations.
> !!! This implies that there was variability in the test environment that
> !!! must be investigated before going further.
> !!! Confidence intervals: Throughput : 13.4%
> !!! Local CPU util : 11.5%
> !!! Remote CPU util : 294.0%
>
> 87380 16384 16384 30.08 668.97 6.79 7.66 1.668 7.879
> !!! WARNING
> !!! Desired confidence was not achieved within the specified iterations.
> !!! This implies that there was variability in the test environment that
> !!! must be investigated before going further.
> !!! Confidence intervals: Throughput : 10.4%
> !!! Local CPU util : 12.7%
> !!! Remote CPU util : 290.8%
>
> 87380 16384 16384 30.10 683.27 6.79 7.68 1.631 7.436
> !!! WARNING
> !!! Desired confidence was not achieved within the specified iterations.
> !!! This implies that there was variability in the test environment that
> !!! must be investigated before going further.
> !!! Confidence intervals: Throughput : 15.0%
> !!! Local CPU util : 11.4%
> !!! Remote CPU util : 70.2%
>
> 87380 16384 16384 30.12 643.66 6.79 10.14 1.735 10.239
>
> Considering the cpu utilization, can I consider that the test are
> running in parallel?

I wouldn't.  As each CPU utilization represents system-wide CPU util, if 
all the tests were running at the same time, I would expect all the CPU 
utilizations to be very close to one another.  That hasn't happened with 
the remote CPU utilization.

> I increase the length of the test and the number of iterations and I
> still not achieve this interval but they decrease.
> netperf -i 100 -cC -p 17170 -l 60 -P 0 -H $1 -t TCP_STREAM -- -C &
> netperf -i 100 -cC -p 17170 -l 60 -P 0 -H $1 -t TCP_STREAM -- -C &
> netperf -i 100 -cC -p 17170 -l 60 -P 0 -H $1 -t TCP_STREAM -- -C &
> netperf -i 100 -cC -p 17170 -l 60 -P 0 -H $1 -t TCP_STREAM -- -C

netperf will have silently capped the -i option at 30...  If you were 
willing to wait 6000 seconds, I suppose you could go with -i 30 -l 200 
and see how that goes.

> Second results:
> !!! WARNING
> !!! Desired confidence was not achieved within the specified iterations.
> !!! This implies that there was variability in the test environment that
> !!! must be investigated before going further.
> !!! Confidence intervals: Throughput : 4.2%
> !!! Local CPU util : 4.1%
> !!! Remote CPU util : 9.1%
>
> 87380 16384 16384 60.06 642.17 6.58 10.74 1.680 10.985
> !!! WARNING
> !!! Desired confidence was not achieved within the specified iterations.
> !!! This implies that there was variability in the test environment that
> !!! must be investigated before going further.
> !!! Confidence intervals: Throughput : 6.1%
> !!! Local CPU util : 4.0%
> !!! Remote CPU util : 10.4%
>
> 87380 16384 16384 60.08 643.44 6.57 10.73 1.679 10.960
> !!! WARNING
> !!! Desired confidence was not achieved within the specified iterations.
> !!! This implies that there was variability in the test environment that
> !!! must be investigated before going further.
> !!! Confidence intervals: Throughput : 5.1%
> !!! Local CPU util : 4.1%
> !!! Remote CPU util : 9.0%
>
> 87380 16384 16384 60.09 648.31 6.58 10.76 1.665 10.896
> !!! WARNING
> !!! Desired confidence was not achieved within the specified iterations.
> !!! This implies that there was variability in the test environment that
> !!! must be investigated before going further.
> !!! Confidence intervals: Throughput : 5.5%
> !!! Local CPU util : 4.0%
> !!! Remote CPU util : 9.0%
>
> 87380 16384 16384 60.09 651.78 6.57 10.73 1.657 10.817

That is much better.  Still, I've taken to using:

http://www.netperf.org/svn/netperf2/trunk/doc/netperf.html#Using-_002d_002denable_002ddemo

and scripting along the lines of:

http://www.netperf.org/svn/netperf2/trunk/doc/examples/runemomniaggdemo.sh

with post-processing via rrdtool for aggregate measurements.

>
> For the -s -S and -m -M options, what scenario could be a good start in
> order to test their influence?

Pretty much up to you.  There are some ancient scripts under 
http://www.netperf.org/svn/netperf2/trunk/doc/examples/ like the 
tcp_stream_script you could use as a starting point - that script is 
focused on single-instance testing.

happy benchmarking,

rick jones