[netperf-talk] Problem with Bidirectional Test
Ankit Goyal
dangerankit at gmail.com
Wed Jul 15 05:40:36 PDT 2009
Hii,
The board has two PCI-Express interfaces, one with four lanes and one with
one lane; 2.5-Gbit/s full duplex per lane; compliant with PCI-Express base
specification 1.1; configurable as root or end point and 2 Ethernet
10/100/1000-Mbit/s, full-duplex MACs with TCP/IP Acceleration Hardware, QoS,
and Jumbo Frame support, supporting GMII/MII, TBI, RTBI, RGMII, SGMII, SMII
interfaces. Memory access layer (MAL) provides DMA capability to both
Ethernet channels.
when I give this command to see the sender and receiver CPU utilisation:
this is for eth0 connected to comp1
./netperf -H computer1 -t TCP_STREAM -c -C -- -s 32K -S 64K
I get this output:
Receive Send Send Elapsed throughput Utilization
Service Demand
socket socket message time 10^6 bits/sec Send receive
send receive
size size size secs local
local local local
bytes bytes bytes %U
%U us/KB us/KB
65536 65536 65536 10.00 576.59 -1.00 39.08
0.000 5.552
This is for eth1 connected to comp2
./netperf -H computer2 -t TCP_STREAM -c -C -- -s 32K -S 64K
Receive Send Send Elapsed throughput Utilization
Service Demand
socket socket message time 10^6 bits/sec Send receive
send receive
size size size secs local
local local local
bytes bytes bytes %U
%U us/KB us/KB
65536 65536 65536 10.00 503.64 -1.00 13.42
0.000 2.183
Why is the sender local cpu utilisation is in negative and why service
demand send is zero?
i am working on different subnets on the ports and they work gud when i send
data through one port only but when i send data through both the ports
bidirectionally instead of getting doubled they get halved..I have a busybox
v1.2.1 on my board and linux kernel 2.6.30 on my board. I dont have
/etc/sysctl.conf file on my board. There are vsftpd.conf,
hosts.conf,nsswitch.conf and xinetd.conf configuration files only.
Please let me know about the throughput becoming halved
Thanks
On Tue, Jul 14, 2009 at 10:32 PM, Rick Jones <rick.jones2 at hp.com> wrote:
> PLEASE keep this in netperf-talk.
>
> Ankit Goyal wrote:
>
>> Hii,
>> I have two ethernet ports of 1Gbps each on my board ,two PCI-Express
>> connectors,
>>
>
> What number of lanes? PCIe 1.1 or PCIe 2.0 etc etc etc.
>
>> a PCI connector, tcp-ip acceleration too. So I connected one port to one
>> PC and another to a Laptop. So i can see the througput unidirectionally. I
>> have busybox on my board and using windows on other(PC & laptop). I have
>> compiled version of netperf in windows and so can run netperf and netserver.
>> So how can i see the results of bidirectional transfer i.e. data coming out
>> from both ports to laptop and PC and data coming in from laptop and PC to
>> these ports?
>> please tell me some commands which does not use enable burst and
>> tcp_maerts(if possible)
>> because i dont have enable burst configured and tcp_maerts does not
>> work(dont know,maybe i have different compiled versions of netperf in
>> windows)
>>
>
> Shame on you :)
>
> Well, there is no way to run without non-trivial concerns about skew error
> if you have neither maerts nor burst mode. You are left with:
>
> On your board:
> 1) start a TCP_STREAM test directed towards the laptop
> 2) start a TCP_STREAM test directed towards the PC
>
> On your laptop:
> 3) start a TCP_STREAM test directed towards your board
>
> One your PC:
> 4) start a TCP_STREAM test directed towards your board
>
>
> And when i do this test that data coming in to 1 port and data going out
>> from 2nd port,my throughput becomes half of the original unidirectional data
>> transfer from one port, why is this so? I should have got double the
>> throughput? it means there is no advantage of using two ethernet ports..is
>> it so?
>> Please help me out.
>>
>
> I am trying...
>
> What is the CPU utlization on your board during these tests? Perhaps you
> have maxed-out the CPU(s).
>
> What are all the IP addresses involved here?
>
> Are both ports of the board configured into the same IP subnet? If so, and
> you are running Linux on the board, you need to use sysctl to set:
>
> net.ipv4.conf.default.arp_ignore = 1
>
> in something that will persist across reboots (eg /etc/sysctl.conf) and
> reboot, and/or set:
>
> net.ipv4.conf.ethN.arp_ignore = 1
> net.ipv4.conf.ethM.arp_ignore = 1
>
> otherwise, ARP will treat those interfaces as rather interchangable and you
> may not have traffic flow the way you think it will.
>
> It would also be best (IMO) to configure each of the two ports on the board
> into a separate IP subnet (and adjust the laptop and PC accordingly) so you
> have a better idea of what sort of routing decisions the stack is going to
> make.
>
> rick jones
>
> Thanks
>>
>>
>> On Mon, Jul 13, 2009 at 9:59 PM, Rick Jones <rick.jones2 at hp.com <mailto:
>> rick.jones2 at hp.com>> wrote:
>>
>> Ankit Goyal wrote:
>>
>> hii,
>>
>> I am working with dual ethernet port on my board. I have cross
>> compiled netperf 2.4.5 on my board. So when I give ./netserver
>> command:
>>
>> Starting netserver at port no 12865
>> Starting netserver at hostname 0.0.0.0 port 12865 and family
>> AF_UNSPEC
>>
>>
>> But this netserver runs on the eth0 port but I want netserver to
>> run on both ethernet ports.
>>
>>
>> That netserver will run over any port. The port over which tests
>> will run will be influenced by the netperf command lines.
>>
>>
>> So how to make netserver run on both ports simultaneously?
>>
>>
>> Assuming we have eth0 at 1.2.3.4 and eth1 at 2.3.4.5 on the
>> netserver system the first version would be
>>
>> netperf -H 1.2.3.4 ...
>> netperf -H 2.3.4.5 ...
>>
>> If your board is running linux, you may need/want to set the
>> "arp_ignore" (sysctl -a | grep ignore) sysctl to "1" to get linux
>> out of its *very* literal interpretation of the weak end system
>> model. By default, any interface in a linux system will respond to
>> ARP requests for any system-local IP address.
>>
>>
>> and if possible can you tell me some ways to increase the
>> throughput? you
>> told me in previously that you can get 1800Mbps on 1Gig
>> bidirectionaly but i
>> am able to get
>> outbound:560mbps
>> inbound:450mbps
>>
>>
>> wat can be done to make it to 1800 Mbps? I will be very thankful
>> if you help me out man.
>>
>>
>> You will need to see first what the bottleneck happens to be.
>> Attached to this message is some boilerplate I have worked-up that
>> may help.
>>
>> Also, what sort of I/O connection does your dual-port chip have on
>> this board? PCIe? PCI-X? Speeds and feeds?
>>
>> rick jones
>> lets keep this discussion on netperf-talk for the benefit of all.
>>
>>
>> Thanks
>>
>>
>>
>>
>>
>> On Thu, Jul 9, 2009 at 10:30 PM, Rick Jones <rick.jones2 at hp.com
>> <mailto:rick.jones2 at hp.com> <mailto:rick.jones2 at hp.com
>>
>> <mailto:rick.jones2 at hp.com>>> wrote:
>>
>> Ankit Goyal wrote:
>>
>> Thanks a ton guys!!
>> If possible could you tell me that how much max
>> bidirectional
>> throughput I can get on 1Gbps ethernet connection?Can I
>> get more
>> than 1 Gig by changing the drivers and kernel?
>> I know its a very relative question but i will be glad if
>> u help
>> me out!
>>
>>
>> In theory you should be able to see O(1800) megabits/s.
>> Certainly
>> that would be my expectation in this day and age:
>>
>>
>> s7:/home/raj/netperf2_trunk# netperf -H sbs133b1.west. -t
>> TCP_RR -f
>> m -- -r 64K -s 256K -S 256K -m 32K -b 8
>> TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0
>> AF_INET to
>> sbs133b1.west (10.208.1.20) port 0 AF_INET : first burst 8
>> Local /Remote
>> Socket Size Request Resp. Elapsed
>> Send Recv Size Size Time Throughput
>> bytes Bytes bytes bytes secs. 10^6bits/sec
>>
>> 262142 262142 65536 65536 10.00 1837.52
>> 524288 524288
>>
>> whether one will always get that with the paired
>> TCP_STREAM/TCP_MAERTS test is an open question:
>>
>> s7:/home/raj/netperf2_trunk# for i in 1; do netperf -t
>> TCP_STREAM -l
>> 60 -H sbs133b1.west -P 0 & netperf -t TCP_MAERTS -l 60 -H
>> sbs133b1.west -P 0 & done
>> [1] 14619
>> [2] 14620
>> s7:/home/raj/netperf2_trunk#
>> 87380 16384 16384 60.02 713.28
>> 87380 16384 16384 60.01 874.86
>>
>> s7:/home/raj/netperf2_trunk# for i in 1; do netperf -t
>> TCP_STREAM -l
>> 120 -H sbs133b1.west -P 0 & netperf -t TCP_MAERTS -l 120 -H
>> sbs133b1.west -P 0 & done
>> [1] 14621
>> [2] 14622
>> s7:/home/raj/netperf2_trunk#
>> 87380 16384 16384 120.03 626.89
>> 87380 16384 16384 120.01 895.40
>>
>> (FWIW, one of the systems involved there is several (4 or 5?)
>> years
>> old now)
>>
>>
>>
>>
>> Some of my checklist items when presented with assertions of poor
>> network performance, in no particular order:
>>
>> *) Is *any one* CPU on either end of the transfer at or close to 100%
>> utilization? A given TCP connection cannot really take advantage
>> of more than the services of a single core in the system, so
>> average CPU utilization being low does not a priori mean things are
>> OK.
>>
>> *) Are there TCP retransmissions being registered in netstat
>> statistics on the sending system? Take a snapshot of netstat -s -t
>> from just before the transfer, and one from just after and run it
>> through beforeafter from
>> ftp://ftp.cup.hp.com/dist/networking/tools:
>>
>> netstat -s -t > before
>> transfer or wait 60 or so seconds if the transfer was already going
>> netstat -s -t > after
>> beforeafter before after > delta
>>
>> *) Are there packet drops registered in ethtool -S statistics on
>> either side of the transfer? Take snapshots in a matter similar to
>> that with netstat.
>>
>> *) Are there packet drops registered in the stats for the switch(es)
>> being traversed by the transfer? These would be retrieved via
>> switch-specific means.
>>
>> *) What is the latency between the two end points. Install netperf on
>> both sides, start netserver on one side and on the other side run:
>>
>> netperf -t TCP_RR -l 30 -H <remote>
>>
>> and invert the transaction/s rate to get the RTT latency. There
>> are caveats involving NIC interrupt coalescing settings defaulting
>> in favor of throughput/CPU util over latency:
>>
>> ftp://ftp.cup.hp.com/dist/networking/briefs/nic_latency_vs_tput.txt
>>
>> but when the connections are over a WAN latency is important and
>> may not be clouded as much by NIC settings.
>>
>> This all leads into:
>>
>> *) What is the *effective* TCP (or other) window size for the
>> connection. One limit to the performance of a TCP bulk transfer
>> is:
>>
>> Tput <= W(eff)/RTT
>>
>> The effective window size will be the lesser of:
>>
>> a) the classic TCP window advertised by the receiver (the value in
>> the TCP header's window field shifted by the window scaling
>> factor exchanged during connection establishment (why one wants
>> to get traces including the connection establishment...)
>>
>> this will depend on whether/what the receiving application has
>> requested via a setsockopt(SO_RCVBUF) call and the sysctl limits
>> set in the OS. If the application does not call
>> setsockopt(SO_RCVBUF) then the Linux stack will "autotune" the
>> advertised window based on other sysctl limits in the OS.
>>
>> b) the computed congestion window on the sender - this will be
>> affected by the packet loss rate over the connection, hence the
>> interest in the netstat and ethtool stats.
>>
>> c) the quantity of data to which the sending TCP can maintain a
>> reference while waiting for it to be ACKnowledged by the
>> receiver - this will be akin to the classic TCP window case
>> above, but on the sending side, and concerning
>> setsockopt(SO_SNDBUF) and sysctl settings.
>>
>> d) the quantity of data the sending application is willing/able to
>> send at any one time before waiting for some sort of
>> application-level acknowledgement. FTP and rcp will just blast
>> all the data of the file into the socket as fast as the socket
>> will take it. scp has some application-layer "windowing" which
>> may cause it to put less data out onto the connection than TCP
>> might otherwise have permitted. NFS has the maximum number of
>> outstanding requests it will allow at one time acting as a
>> defacto "window" etc etc etc
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.netperf.org/pipermail/netperf-talk/attachments/20090715/62a0355f/attachment-0001.htm
More information about the netperf-talk
mailing list