[netperf-talk] SCTP Benchmark significantly slow

[Chris Lydick]

Vlad and Rick,

Geniuses! It worked.

$ ./netperf -H xx.xx.xx.xx -c 100 -C 100 -t SCTP_STREAM -l 30 -- -s
16777216,16777216 -S 16777216,16777216 -m 8952
Warning! getaddrinfo returned a result which did not match
requested protocol. Kludging. Please contact your vendor for a fix.
Warning! getaddrinfo returned a result which did not match
requested protocol. Kludging. Please contact your vendor for a fix.
SCTP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to xx.xx.xx.xx
(xx.xx.xx.xx) port 0 AF_INET
Recv   Send    Send                          Utilization       Service
Demand
Socket Socket  Message  Elapsed              Send     Recv     Send   
Recv
Size   Size    Size     Time     Throughput  local    remote   local  
remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB  
us/KB

33554432 33554432   8952    30.02       990.56   99.97    59.39    8.267
  4.912

Thanks for your help.

Chris Lydick


Quoting Rick Jones <rick.jones2 at hp.com>:

> Vlad Yasevich wrote:
> > Chris Lydick wrote:
> >
> >>So, it does look like TSO and CKO are both being utilized for TCP
> like
> >>you said, Rick.
> >
> >
> > Well, since the cards don't have any support for SCTP, TCP gets the
> > advantage  there.
> >
> >
> >>As for the memory, here's that info:
> >>
> >>$ RMEM_DEF=`cat /proc/sys/net/core/rmem_default`; echo $RMEM_DEF
> >>129024 (it was the same for wmem_default)
> >>
> >>$ RMEM_MAX=`cat /proc/sys/net/core/rmem_max`; echo $RMEM_MAX
> >>16777216 (again, same for wmem_max)
> >>
> >>Now, here's the output that appears when I'm running netperf. Mind
> you,
> >>on the other end of things (at server host), the command I ran was
> >>"netserver -4".
> >>
> >>$ LOC_RATE=`./netperf -t REM_CPU`; echo $LOC_RATE
> >>100
> >>$ REM_RATE=`./netperf -H xxx.xxx.xxx.xxx -t REM_CPU`; echo
> $REM_RATE
> >>100
> >>$ ./netperf -H xxx.xxx.xxx.xxx -c $LOC_RATE -C $REM_RATE -t
> SCTP_STREAM
> >>-l 30 -a $RMEM_MAX,$RMEM_MAX -A $RMEM_MAX,$RMEM_MAX
> >
> >
> > why are you specifying the alignment of 16777216?
>
> That could be having some rather significant consequences on the
> buffer
> allocation being done by netperf.  It will add that value to the size
> of
> the buffers it allocates, and since it sends from a ring of buffers,
> which has as many entries as SO_[SND|RCV]BUF size divided by the
> send/recv size plus one entry you are likely asking netperf to be
> skipping all over the place cache-miss wise.  The default alignement
> for
> a netperf buffer is 8 bytes, which (at least at the time) was
> sufficient
> for optimal copy routines.
> >
> > Let's do some math:
> >
> > mtu = 9000.
> > largest sctp data without fragments (optimum result):
> > 	9000 - 20 - 12 (sctp hdr) - 16 (data hdr) = 8952
> >
> > That's your ideal message size.  Remember that SCTP is a message
> oriented
> > protocol that preserves message boundaries.
> >
> > So try running:
> > netperf -H xxx.xxx.xxx.xxx -c $LOC_RATE - C $REM_RATE -t
> SCTP_STREAM \
> > -l 30 -- -s $RMEM_MAX,$RMEM_MAX -S $RMEM_MAX,$RMEM_MAX -m 8952
> >
> > (I think I got that right)
>
> Replacing the global -a/-A with test-specific -s/-S is almost
> certainly
> the right thing to do.
>
> FWIW, the CPU util mechanism on Linux does not require calibration,
> so
> while stuff cribbed from the increasingly ancien example scripts will
> still work, the $REM_RATE $LOC_RATE stuff isn't strictly necessary to
> maintain quick measurement times.  Doesn't hurt really, as if you are
> on
> a reasonably current netperf (forget which) the calibration-free CPU
> util mechanisms will have the REM_CPU or LOC_CPU tests finish in very
> much less than the traditional 40 seconds.
>
> >
> >>Here's another issue... When running a utility I found a while back
> >>called SCTPPerf (http://tinyurl.com/34hcwu), I was able to get
> better
> >>results... My best throughput test was using the following
> parameters.
> >>While it seems I could use this program, I'd much rather work with
> >>Netperf. Any suggestions on this one? *Note* message length is 8938
> >>because that plus lower-layer headers = 9000 bytes, the MTU.
> >>
> >>(server) $ ./scptperf_srv -P 6420 -H xxx.xxx.xxx.xxx -f 16384 -v
> >>(client) $ ./scptperf_clnt -P 6421 -p 6420 -H xxx.xxx.xxx.xxx -h
> >>xxx.xxx.xxx.xxx -l 8938 -f 16384 -t 1 -v -m 1 -x 1
>
> I take it there are "SCTP calls" one can make to set congestion
> window
> parameters (-f)?
>
> The URL was somewhat thin on units, but I'm guessing that the test
> ran
> for only one second? Seems confirmed by the output below.  Might be
> enough, but I'd be more comfortable with a longer run time.
>
> One other thing that netperf does is make certain the remote has
> received all the data - and I am pretty sure that carried over into
> the
> netperf SCTP tests.  In nettest_bsd.c for say send_tcp_stream() you
> can
> see the call to shutdown() followed by the call to recv() to get the
> indication that the remote has seen our close and has drained all the
> data.  It would be good, especially for very short runs, to make sure
> that this SCTPperf tool does something similar.
>
> >>
> >>Local host:               xxx.xxx.xxx.xxx
> >>Local port:               6421
> >>Remote host:              xxx.xxx.xxx.xxx
> >>Remote port:              6420
> >>Packet size:              8938
> >>Measure period:           1
> >>Print period:             1
> >>Streams:                  1
> >>Sender buffer:            16777216
> >>Receiver buffer:          16777216
> >>Verbose:                  1
> >>
> >>Binding single address
> >>
> >>[2007/06/01 19:35:15] 1s_BDW = 873.425110 Mbits/s, PACKETS = 12809,
> TIME
> >>= 1 sec 46 usec
> >>
> >>BEGIN:              2007/06/01 19:35:14
> >>END:                2007/06/01 19:35:15
> >>TIME:               1 sec 46 usec
> >>MESSAGE SIZE:       8938 bytes
> >>SENT:               12809 messages
> >>DATA SENT:          114486842 bytes
> >>BANDWIDTH:          873.425110 Mbits/s
> >>
> >>Are there any alternate libraries for SCTP I might be able to
> download
> >>and try? Or are the libraries I included in the first email
> sufficient?
> >>
> >
> >
> > It's not the issue of libraries.  The libraries are just a tiny
> shim layer
> > that lets you write apps easier instead of messing around with all
> the
> > sendmsg() stuff.
> >
> >
> >>I'm ready to start digging through the source code on both programs
> and
> >>see if there are different SCTP calls between SCTPPerf and Netperf.
> >>Looking forward to a reply :)
>
> By all means start digging.  Netperf would love additional input on
> the
> tests.
>
> > You can also try iperf.  That also has SCTP support.
> >
> > My guess the difference in results in the difference in message
> sizes that are
> > used by each application.  What you want to avoid is any semblance
> of SCTP level
> > fragmentation as that will always slow things down.
> >
> > -vlad
>
>