[netperf-talk] SCTP Benchmark significantly slow

[Rick Jones]

Vlad Yasevich wrote:
> Chris Lydick wrote:
> 
>>So, it does look like TSO and CKO are both being utilized for TCP like
>>you said, Rick.
> 
> 
> Well, since the cards don't have any support for SCTP, TCP gets the
> advantage  there.
> 
> 
>>As for the memory, here's that info:
>>
>>$ RMEM_DEF=`cat /proc/sys/net/core/rmem_default`; echo $RMEM_DEF
>>129024 (it was the same for wmem_default)
>>
>>$ RMEM_MAX=`cat /proc/sys/net/core/rmem_max`; echo $RMEM_MAX
>>16777216 (again, same for wmem_max)
>>
>>Now, here's the output that appears when I'm running netperf. Mind you,
>>on the other end of things (at server host), the command I ran was
>>"netserver -4".
>>
>>$ LOC_RATE=`./netperf -t REM_CPU`; echo $LOC_RATE
>>100
>>$ REM_RATE=`./netperf -H xxx.xxx.xxx.xxx -t REM_CPU`; echo $REM_RATE
>>100
>>$ ./netperf -H xxx.xxx.xxx.xxx -c $LOC_RATE -C $REM_RATE -t SCTP_STREAM
>>-l 30 -a $RMEM_MAX,$RMEM_MAX -A $RMEM_MAX,$RMEM_MAX
> 
> 
> why are you specifying the alignment of 16777216?

That could be having some rather significant consequences on the buffer 
allocation being done by netperf.  It will add that value to the size of 
the buffers it allocates, and since it sends from a ring of buffers, 
which has as many entries as SO_[SND|RCV]BUF size divided by the 
send/recv size plus one entry you are likely asking netperf to be 
skipping all over the place cache-miss wise.  The default alignement for 
a netperf buffer is 8 bytes, which (at least at the time) was sufficient 
for optimal copy routines.
> 
> Let's do some math:
> 
> mtu = 9000.
> largest sctp data without fragments (optimum result):
> 	9000 - 20 - 12 (sctp hdr) - 16 (data hdr) = 8952
> 
> That's your ideal message size.  Remember that SCTP is a message oriented
> protocol that preserves message boundaries.
> 
> So try running:
> netperf -H xxx.xxx.xxx.xxx -c $LOC_RATE - C $REM_RATE -t SCTP_STREAM \
> -l 30 -- -s $RMEM_MAX,$RMEM_MAX -S $RMEM_MAX,$RMEM_MAX -m 8952
> 
> (I think I got that right)

Replacing the global -a/-A with test-specific -s/-S is almost certainly 
the right thing to do.

FWIW, the CPU util mechanism on Linux does not require calibration, so 
while stuff cribbed from the increasingly ancien example scripts will 
still work, the $REM_RATE $LOC_RATE stuff isn't strictly necessary to 
maintain quick measurement times.  Doesn't hurt really, as if you are on 
a reasonably current netperf (forget which) the calibration-free CPU 
util mechanisms will have the REM_CPU or LOC_CPU tests finish in very 
much less than the traditional 40 seconds.

> 
>>Here's another issue... When running a utility I found a while back
>>called SCTPPerf (http://tinyurl.com/34hcwu), I was able to get better
>>results... My best throughput test was using the following parameters.
>>While it seems I could use this program, I'd much rather work with
>>Netperf. Any suggestions on this one? *Note* message length is 8938
>>because that plus lower-layer headers = 9000 bytes, the MTU.
>>
>>(server) $ ./scptperf_srv -P 6420 -H xxx.xxx.xxx.xxx -f 16384 -v
>>(client) $ ./scptperf_clnt -P 6421 -p 6420 -H xxx.xxx.xxx.xxx -h
>>xxx.xxx.xxx.xxx -l 8938 -f 16384 -t 1 -v -m 1 -x 1

I take it there are "SCTP calls" one can make to set congestion window 
parameters (-f)?

The URL was somewhat thin on units, but I'm guessing that the test ran 
for only one second? Seems confirmed by the output below.  Might be 
enough, but I'd be more comfortable with a longer run time.

One other thing that netperf does is make certain the remote has 
received all the data - and I am pretty sure that carried over into the 
netperf SCTP tests.  In nettest_bsd.c for say send_tcp_stream() you can 
see the call to shutdown() followed by the call to recv() to get the 
indication that the remote has seen our close and has drained all the 
data.  It would be good, especially for very short runs, to make sure 
that this SCTPperf tool does something similar.

>>
>>Local host:               xxx.xxx.xxx.xxx
>>Local port:               6421
>>Remote host:              xxx.xxx.xxx.xxx
>>Remote port:              6420
>>Packet size:              8938
>>Measure period:           1
>>Print period:             1
>>Streams:                  1
>>Sender buffer:            16777216
>>Receiver buffer:          16777216
>>Verbose:                  1
>>
>>Binding single address
>>
>>[2007/06/01 19:35:15] 1s_BDW = 873.425110 Mbits/s, PACKETS = 12809, TIME
>>= 1 sec 46 usec
>>
>>BEGIN:              2007/06/01 19:35:14
>>END:                2007/06/01 19:35:15
>>TIME:               1 sec 46 usec
>>MESSAGE SIZE:       8938 bytes
>>SENT:               12809 messages
>>DATA SENT:          114486842 bytes
>>BANDWIDTH:          873.425110 Mbits/s
>>
>>Are there any alternate libraries for SCTP I might be able to download
>>and try? Or are the libraries I included in the first email sufficient?
>>
> 
> 
> It's not the issue of libraries.  The libraries are just a tiny shim layer
> that lets you write apps easier instead of messing around with all the
> sendmsg() stuff.
> 
> 
>>I'm ready to start digging through the source code on both programs and
>>see if there are different SCTP calls between SCTPPerf and Netperf.
>>Looking forward to a reply :)

By all means start digging.  Netperf would love additional input on the 
tests.

> You can also try iperf.  That also has SCTP support.
> 
> My guess the difference in results in the difference in message sizes that are
> used by each application.  What you want to avoid is any semblance of SCTP level
> fragmentation as that will always slow things down.
> 
> -vlad