[netperf-talk] SCTP Benchmark significantly slow
Chris Lydick
lydick at ksu.edu
Fri Jun 1 12:47:49 PDT 2007
Thanks both of you for your input. Here is some more data (which may or
may not prove to be useful).
$ sudo ethtool -k eth3
Offload parameters for eth3:
Cannot get device udp large send offload settings: Operation not
supported
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: off
So, it does look like TSO and CKO are both being utilized for TCP like
you said, Rick.
The more generic ethtool output (don't know if this will prove useful):
$ sudo ethtool eth3
Settings for eth3:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 0
Transceiver: external
Auto-negotiation: on
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
Link detected: yes
$ netstat -i
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP
TX-OVR Flg
eth1 9000 0 2137322 0 0 0 1962420 0 0
0 BMRU
eth3 9000 0 41296759 0 2598 0 23892853 0 0
0 BMRU
lo 16436 0 1896254 0 0 0 1896254 0 0
0 LRU
and running beforeafter, I find that between the runs...:
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP
TX-OVR Flg
eth1 0 0 1023 0 0 0 241 0 0 0 BMRU
eth3 0 0 243 0 0 0 402 0 0 0 BMRU
lo 0 0 2 0 0 0 2 0 0 0 LRU
..there aren't any apparent errors.
As for the memory, here's that info:
$ RMEM_DEF=`cat /proc/sys/net/core/rmem_default`; echo $RMEM_DEF
129024
(it was the same for wmem_default)
$ RMEM_MAX=`cat /proc/sys/net/core/rmem_max`; echo $RMEM_MAX
16777216
(again, same for wmem_max)
Now, here's the output that appears when I'm running netperf. Mind you,
on the other end of things (at server host), the command I ran was
"netserver -4".
$ LOC_RATE=`./netperf -t REM_CPU`; echo $LOC_RATE
100
$ REM_RATE=`./netperf -H xxx.xxx.xxx.xxx -t REM_CPU`; echo $REM_RATE
100
$ ./netperf -H xxx.xxx.xxx.xxx -c $LOC_RATE -C $REM_RATE -t SCTP_STREAM
-l 30 -a $RMEM_MAX,$RMEM_MAX -A $RMEM_MAX,$RMEM_MAX
Warning! getaddrinfo returned a result which did not match
requested protocol. Kludging. Please contact your vendor for a fix.
Warning! getaddrinfo returned a result which did not match
requested protocol. Kludging. Please contact your vendor for a fix.
SCTP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 134.253.7.209
(134.253.7.209) port 0 AF_INET
Recv Send Send Utilization Service
Demand
Socket Socket Message Elapsed Send Recv Send
Recv
Size Size Size Time Throughput local remote local
remote
bytes bytes bytes secs. 10^6bits/s % S % S us/KB
us/KB
129024 129024 129024 30.21 2.60 0.09 0.02 2.730
0.596
Here's another issue... When running a utility I found a while back
called SCTPPerf (http://tinyurl.com/34hcwu), I was able to get better
results... My best throughput test was using the following parameters.
While it seems I could use this program, I'd much rather work with
Netperf. Any suggestions on this one? *Note* message length is 8938
because that plus lower-layer headers = 9000 bytes, the MTU.
(server) $ ./scptperf_srv -P 6420 -H xxx.xxx.xxx.xxx -f 16384 -v
(client) $ ./scptperf_clnt -P 6421 -p 6420 -H xxx.xxx.xxx.xxx -h
xxx.xxx.xxx.xxx -l 8938 -f 16384 -t 1 -v -m 1 -x 1
Local host: xxx.xxx.xxx.xxx
Local port: 6421
Remote host: xxx.xxx.xxx.xxx
Remote port: 6420
Packet size: 8938
Measure period: 1
Print period: 1
Streams: 1
Sender buffer: 16777216
Receiver buffer: 16777216
Verbose: 1
Binding single address
[2007/06/01 19:35:15] 1s_BDW = 873.425110 Mbits/s, PACKETS = 12809, TIME
= 1 sec 46 usec
BEGIN: 2007/06/01 19:35:14
END: 2007/06/01 19:35:15
TIME: 1 sec 46 usec
MESSAGE SIZE: 8938 bytes
SENT: 12809 messages
DATA SENT: 114486842 bytes
BANDWIDTH: 873.425110 Mbits/s
Are there any alternate libraries for SCTP I might be able to download
and try? Or are the libraries I included in the first email sufficient?
I'm ready to start digging through the source code on both programs and
see if there are different SCTP calls between SCTPPerf and Netperf.
Looking forward to a reply :)
Looks like I may be keeping you guys busy!
-Chris
Quoting Vlad Yasevich <vladislav.yasevich at hp.com>:
> Thanks to rick for forwarding this.
>
> Rick Jones wrote:
> > Chris Lydick wrote:
> >> The only issue is that for some reason, I am unable to get any
> "decent"
> >> results using the SCTP_STREAM test within Netperf. Here are some
> of my
> >> findings:
> >>
> >> Using a 1Gbps Ethernet line between two nodes, I receive nearly
> >> 987Mb/sec through the TCP_STREAM (this seems very accurate),
> >
> > If you have JumboFrames enabled at least.
> >
> >> while on
> >> the SCTP_STREAM side of things, I am only receiving at most 2.4
> Mb/sec.
> >>
>
> That's close to what I can get. I can squeeze out ~3mb/sec with the
> latest
> kernel sources.
>
> SCTP is frightfully inefficient and has not gone through a full
> performance
> analysis. The biggest roadblock is the use memory copies. There are
> at
> least 2 memory copies for every data message. If the message is
> bigger then
> the mtu, there will be more memory copies.
>
> Second, the receive buffer management is crufty and prevents us from
> fully utilizing
> our congestion window. This is being addressed though.
>
> There are other areas as well.
>
> >> I notice similar behavior when using one machine as both the
> source and
> >> destination of the test.
> >>
>
> You should get better results over loopback since we don't do
> checksumming there.
>
> >> I am using two Linux (Ubuntu 7.04) machines for the benchmarking,
> >> equipped with libsctp1 (1.0.6.dfsg-4), and libsctp-dev
> (1.0.6.dfsg-4)
> >> libraries.
>
> This is fine, but the kernel might not have all the latest bits.
> Don't know
> how well Ubuntu tracks upstream patches.
>
> >>
> >> I have also tried adjusting the buffers through the command line
> >> prompts, but they have had little effect.
> >>
> >> Do you have any suggestions?
>
> Max out the buffers in the kernel ([r/w]mem_default and
> [r/w]mem_max).
> The _max thresholds limit what the user can specify.
>
> >
> > Some additional specifics wrt parameters being passed on the
> netperf
> > command line would I suspect help :)
> >
> > Other things which are often useful include:
> >
> > *) CPU utilization, while netperf doesn't report it, per-CPU util
> might
> > be nice too if this is between MP systems. Most GbE NICs can do
> CKO and
> > TSO, which will help TCP but will do zip for SCTP.
> >
> > *) Checking ethtool stats for losses
> >
> > *) Checking netstat stats for losses (I'm ass-u-me-ing there are
> sctp
> > stats)
>
> What Rick said.
>
> -vlad
>
>
More information about the netperf-talk
mailing list