[netperf-talk] Help me to understand the results

[Rick Jones]

Steevan Rodrigues wrote:
> Hi Rick,
> 
> Thanks for the details.
> Can I consider the result even when there is a warning about confidence 
> level ?
> Will  that lead to inaccuracy ?

The warning about confidence interval is just that, a warning.  It suggests 
there is greater variability than one *might* want.  It simply means the error 
bars are rather wide.

So, if netperf reports 60% for remote CPU util, and says the width of the 
confidence interval is 50%, and one asked for 99% confidence level, it means 
that netperf cannot state that it is 99% confident that the actual mean remote 
CPU utilization was anything other than somewhere between 45 and 75% (ie +/- 25% 
of the reported mean.

> Also what happens when I use -r options (i.e  Allow confidence to be hit 
> on result only) ?
> Is it more accurate or is to ignore confidence warning ?

If you use -r to apply confidence to result only, it means just that.  And you 
will not have (iirc) an idea of how variable the local or remote CPU utilization 
happens to have been.

In the omni tests, I don't emit warnings at all, but simply report the final 
width and leave it entirely to the consumer to decide if it is important.

happy benchmarking,

rick jones

> 
> Thanks,
> Steev
> 
> 
> On Thu, Jan 13, 2011 at 11:58 PM, Rick Jones <rick.jones2 at hp.com 
> <mailto:rick.jones2 at hp.com>> wrote:
> 
>     Steevan Rodrigues wrote:
> 
>         Hi,
> 
>         I am newbie to netperf.
>         I ran following command
>         $ /usr/local/bin/netperf -l 5 -H 192.168.11.207 -t TCP_STREAM -c
>         LOC_CPU -C REM_CPU -i 10,2 -I 95,3 -- -m 1 -s 32768 -S 32768
> 
>         And obtained following output (please see below). Can you please
>         explain me following ?
>         1) Why there is a warning about desired "Confidence level" ?
> 
> 
>     http://www.netperf.org/svn/netperf2/trunk/doc/netperf.html#index-g_t_002dI_002c-Global-26
> 
> 
>         2) When I include -c LOC_CPU  -C REM_CPU  I get  details about
>         "utilization" and "Service Demand" . What do they represent ?
>         Do they represent local and remote CPU utilization ?
> 
> 
>     I suspect there has been some cut-and-paste confusion from a script
>     somewhere as I would expect that to be $LOC_CPU and $REM_CPU.
>      Regardless the -c option requests measurement of local CPU
>     utilization, -C requests measurement of remote CPU utilization.
> 
>     http://www.netperf.org/svn/netperf2/trunk/doc/netperf.html#CPU-Utilization
>     http://www.netperf.org/svn/netperf2/trunk/doc/netperf.html#index-g_t_002dc_002c-Global-18
> 
> 
>         3) How max and min number of iterations are actually used ?
> 
> 
>     See the URL in the answer to 1.
> 
> 
>         4) Actually I want to measure throughput vs CPU Load for my system.
>           What is the best command for the above purpose ?
> 
> 
>     Are you trying to say you want to measure what throughput can be
>     achieved at discrete levels of CPU utilization?   That may be
>     something of a challenge.
> 
>     It may be possible to measure CPU utilization at a given throughput
>     level via:
> 
>     http://www.netperf.org/svn/netperf2/trunk/doc/netperf.html#index-g_t_002dw_002c-Global-39
>     http://www.netperf.org/svn/netperf2/trunk/doc/netperf.html#index-g_t_002db_002c-Global-16
> 
>     However, achieving a given level of throughput can be accomplished
>     with more than one set of values for -b and -w, which may affect the
>     CPU utilization. Toss-in the effect of the various values of the -m
>     test-specific option and one is really talking about a rather large
>     space.
> 
> 
>         5) Do I need to use a crossover cable for the test setup ?
> 
> 
>     1GbE and later networking does not require a cross-over cable.
>      100BT and below does.
> 
> 
>         6) Can I have multiple servers and one client ? Or multiple
>         client and one server setup ?
> 
> 
>     http://www.netperf.org/svn/netperf2/trunk/doc/netperf.html#Using-Netperf-to-Measure-Aggregate-Performance
> 
> 
>         ==================================================
>         TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
>         192.168.11.207 (192.168.11.207) port 0 AF_INET : +/-1.500% @ 95%
>         conf.
>         !!! WARNING
>         !!! Desired confidence was not achieved within the specified
>         iterations.
>         !!! This implies that there was variability in the test
>         environment that
>         !!! must be investigated before going further.
>         !!! Confidence intervals: Throughput      : 1.164%
>         !!!                       Local CPU util  : 0.000%
>         !!!                       Remote CPU util : 56.955%
> 
>         Recv   Send     Send                                        
>         Utilization       Service Demand
>         Socket Socket  Message  Elapsed                     Send    
>         Recv     Send    Recv
>         Size     Size     Size           Time     Throughput    local  
>          remote   local   remote
>         bytes    bytes   bytes         secs.    10^6bits/s      % S    
>          % S      us/KB   us/KB
> 
>          65536  65536      1             5.00          9.46      
>          100.00   59.60    865.994  516.212
>         ==============================================================
> 
>         Thanks for your help,
>         Steevan
> 
> 
>         ------------------------------------------------------------------------
> 
>         _______________________________________________
>         netperf-talk mailing list
>         netperf-talk at netperf.org <mailto:netperf-talk at netperf.org>
>         http://www.netperf.org/cgi-bin/mailman/listinfo/netperf-talk
> 
> 
>