[netperf-talk] FreeBSD sysctl cpu utilization diffs

[Rick Jones]

Andrew Gallatin wrote:
> Rick Jones writes:
>  > If the mechanism is what I think it was, basically, the idea was to 
>  > compare how fast something increments when the system is "idle" vs when 
>  > the system is running netperf.  Hence the calibration period.
> 
> The clock will always tick at stathz, so the calibration is not needed.
> The FreeBSD people are very picky about accurate timekeeping and
> accounting.

Good.

>  > That just shows they haven't gotten to very high CPU counts yet :)
> 
> Yeah, I imagine they'll keep it per-cpu and add it at sysctl
> time eventually.  They've apparently ported to ontario/niagara,
> so that's 32 CPUs.

Well, the appearance of 32 CPUs.  How one handles "CPU utilization" in 
the context of HW threading is a very "interesting" topic - one I'm 
running into for AIX with netperf4.

> 
> 
>  > >  See the statclock() routine in sys/kern/kern_clock.c
>  > > (http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/kern_clock.c?rev=1.187&content-type=text/x-cvsweb-markup)
>  > > 
>  > > The patch simply takes a snapshot of the counters when the test
>  > > starts, another snapshot when the test ends, and caclulates the 
>  > > CPU utilization by figuring out how much idle time there was as
>  > > compared to the other, non-idle CPU states.
>  > > 
>  > > I have tested my patch on FreeBSD 5.4 i386 UP, 6.0 i386 SMP, and
>  > > 7.0 amd64 SMP and the CPU time reported by netperf agrees with
>  > > the CPU time reported by the FreeBSD CPU monitoring tools (vmstat,
>  > > systat -vm, iostat).
>  > 
>  > Since they are likely using the same mechanism I would hope so :)
> 
> Just wanted to show that I've tested on 64/32 bit and SMP/up..

Fair enough :)

>  > What does it look like compared to the looper method?  One of the 
>  > concerns I have about all these "statistical" sampling methods is 
>  > whether or not they remain accurate under a very high interrupt load 
>  > such as that produced by netperf.
> 
> Looper does not seem to compile in 2.4.1, so it is hard to say. :(

I think that is resolved in the SVN top of trunk:

http://www.netperf.org/svn/netperf2/trunk/

There will be some effect on the measurment simply because there will be 
N processes sitting out there spinning madly, but the differences should 
be _reasonably_ small.  If the CPU util is very different then we may 
need to start talking to those picky FreeBSD timekeeping folks :)

I'll go ahead and apply your patch to the TOT, based on your assertion 
of the pickiness of the FreeBSD timekeepers :)  Working calibration-free 
mechanisms in netperf2 is a good thing - makes it easier to do the same 
in netperf4.

BTW, _in_ netperf4 you will see a "calibration" field in the 
netsysstats_mumble.c files - that should be set to have however many 
ticks there _should_ be over the interval and not simply a sum of the 
ticks that were returned by the likes of sysctl/pstat/whatever.  Don't 
let the "elapsed" calculation in the platform-specific netsysstats 
confuse you - the netsysstats_common.c stuff will be taking deltas to 
have it all do the right thing.

NB That is why in netperf4 all those counters need to remain 64-bit 
quantities, regardless of the bitness of the binary. (he said in 
reference to your netperf2 patch's changing things from uint64_t to long :)

> BTW, should I be using netperf4 from svn and submitting patches against
> that?  Or is there still life in the 2.x series?

There is still life in the 2.X series.  Given the rather large "basic 
assumtpions" differences between netperf2 and netperf4 I have no plans 
to retire netperf2 any time soon.

When you do netperf4 work, please do it against the glib_migration branch:

http://www.netperf.org/svn/netperf4/branches/glib_migration

as that will before toooo much longer become what is in the trunk.  I'm 
only waiting long enough for my partner-in-crime in the UX networking 
lab to get some additional stuff added to nettest_bsd.c on the mainline 
before I force him to migrate all his UX systems to include glib :)

> 
> BTW, I *love* the CPU binding for linux in 2.4!

I finally bit the bullet and agreed to accept CPU binding for non-looper 
processes - it should be there for most platforms now.  It is also in 
the netperf4 glib_migration branch.  A bit more sophisiticated now - we 
launch test threads from the same CPU to which they are to be bound 
(allow stack allocation to be based on the CPU to which the test thread 
will be bound)

happy benchmarking,

rick jones