Hii,<br><br>The board has two PCI-Express interfaces, one with four lanes and one with one lane;
2.5-Gbit/s full duplex per lane; compliant with PCI-Express base
specification 1.1; configurable as root or end point and 2 Ethernet 10/100/1000-Mbit/s, full-duplex MACs with TCP/IP
Acceleration Hardware, QoS, and Jumbo Frame support, supporting
GMII/MII, TBI, RTBI, RGMII, SGMII, SMII interfaces. Memory access layer
(MAL) provides DMA capability to both Ethernet channels.<br><br>when I give this command to see the sender and receiver CPU utilisation:<br><br>this is for eth0 connected to comp1<br>./netperf -H computer1 -t TCP_STREAM -c -C -- -s 32K -S 64K<br>
<br>I get this output:<br><br>Receive Send Send Elapsed throughput Utilization Service Demand<br>socket socket message time 10^6 bits/sec Send receive send receive<br>size size size secs local local local local<br>
bytes bytes bytes %U %U us/KB us/KB<br><br>65536 65536 65536 10.00 576.59 -1.00 39.08 0.000 5.552<br><br>This is for eth1 connected to comp2<br>
<br>./netperf -H computer2 -t TCP_STREAM -c -C -- -s 32K -S 64K<br><br>
Receive Send Send Elapsed throughput Utilization Service Demand<br>
socket socket message time 10^6 bits/sec Send receive send receive<br>
size size size secs local local local local<br>
bytes bytes bytes %U %U us/KB us/KB<br>
<br>
65536 65536 65536 10.00 503.64 -1.00 13.42 0.000 2.183<br>
<br><br>Why is the sender local cpu utilisation is in negative and why service demand send is zero?<br><br><br>i am working on different subnets on the ports and they work gud when i send data through one port only but when i send data through both the ports bidirectionally instead of getting doubled they get halved..I have a busybox v1.2.1 on my board and linux kernel 2.6.30 on my board. I dont have /etc/sysctl.conf file on my board. There are vsftpd.conf, hosts.conf,nsswitch.conf and xinetd.conf configuration files only.<br>
<br>Please let me know about the throughput becoming halved<br><br>Thanks<br><br><div class="gmail_quote">On Tue, Jul 14, 2009 at 10:32 PM, Rick Jones <span dir="ltr"><<a href="mailto:rick.jones2@hp.com">rick.jones2@hp.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">PLEASE keep this in netperf-talk.<div class="im"><br>
<br>
Ankit Goyal wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hii,<br>
I have two ethernet ports of 1Gbps each on my board ,two PCI-Express connectors, <br>
</blockquote>
<br></div>
What number of lanes? PCIe 1.1 or PCIe 2.0 etc etc etc.<div class="im"><br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
a PCI connector, tcp-ip acceleration too. So I connected one port to one PC and another to a Laptop. So i can see the througput unidirectionally. I have busybox on my board and using windows on other(PC & laptop). I have compiled version of netperf in windows and so can run netperf and netserver. So how can i see the results of bidirectional transfer i.e. data coming out from both ports to laptop and PC and data coming in from laptop and PC to these ports?<br>
please tell me some commands which does not use enable burst and tcp_maerts(if possible)<br>
because i dont have enable burst configured and tcp_maerts does not work(dont know,maybe i have different compiled versions of netperf in windows)<br>
</blockquote>
<br></div>
Shame on you :)<br>
<br>
Well, there is no way to run without non-trivial concerns about skew error if you have neither maerts nor burst mode. You are left with:<br>
<br>
On your board:<br>
1) start a TCP_STREAM test directed towards the laptop<br>
2) start a TCP_STREAM test directed towards the PC<br>
<br>
On your laptop:<br>
3) start a TCP_STREAM test directed towards your board<br>
<br>
One your PC:<br>
4) start a TCP_STREAM test directed towards your board<div class="im"><br>
<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
And when i do this test that data coming in to 1 port and data going out from 2nd port,my throughput becomes half of the original unidirectional data transfer from one port, why is this so? I should have got double the throughput? it means there is no advantage of using two ethernet ports..is it so?<br>
Please help me out.<br>
</blockquote>
<br></div>
I am trying...<br>
<br>
What is the CPU utlization on your board during these tests? Perhaps you have maxed-out the CPU(s).<br>
<br>
What are all the IP addresses involved here?<br>
<br>
Are both ports of the board configured into the same IP subnet? If so, and you are running Linux on the board, you need to use sysctl to set:<br>
<br>
net.ipv4.conf.default.arp_ignore = 1<br>
<br>
in something that will persist across reboots (eg /etc/sysctl.conf) and reboot, and/or set:<br>
<br>
net.ipv4.conf.ethN.arp_ignore = 1<br>
net.ipv4.conf.ethM.arp_ignore = 1<br>
<br>
otherwise, ARP will treat those interfaces as rather interchangable and you may not have traffic flow the way you think it will.<br>
<br>
It would also be best (IMO) to configure each of the two ports on the board into a separate IP subnet (and adjust the laptop and PC accordingly) so you have a better idea of what sort of routing decisions the stack is going to make.<br>
<br>
rick jones<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Thanks<div><div></div><div class="h5"><br>
<br>
On Mon, Jul 13, 2009 at 9:59 PM, Rick Jones <<a href="mailto:rick.jones2@hp.com" target="_blank">rick.jones2@hp.com</a> <mailto:<a href="mailto:rick.jones2@hp.com" target="_blank">rick.jones2@hp.com</a>>> wrote:<br>
<br>
Ankit Goyal wrote:<br>
<br>
hii,<br>
<br>
I am working with dual ethernet port on my board. I have cross<br>
compiled netperf 2.4.5 on my board. So when I give ./netserver<br>
command:<br>
<br>
Starting netserver at port no 12865<br>
Starting netserver at hostname 0.0.0.0 port 12865 and family<br>
AF_UNSPEC<br>
<br>
<br>
But this netserver runs on the eth0 port but I want netserver to<br>
run on both ethernet ports.<br>
<br>
<br>
That netserver will run over any port. The port over which tests<br>
will run will be influenced by the netperf command lines.<br>
<br>
<br>
So how to make netserver run on both ports simultaneously?<br>
<br>
<br>
Assuming we have eth0 at 1.2.3.4 and eth1 at 2.3.4.5 on the<br>
netserver system the first version would be<br>
<br>
netperf -H 1.2.3.4 ...<br>
netperf -H 2.3.4.5 ...<br>
<br>
If your board is running linux, you may need/want to set the<br>
"arp_ignore" (sysctl -a | grep ignore) sysctl to "1" to get linux<br>
out of its *very* literal interpretation of the weak end system<br>
model. By default, any interface in a linux system will respond to<br>
ARP requests for any system-local IP address.<br>
<br>
<br>
and if possible can you tell me some ways to increase the<br>
throughput? you<br>
told me in previously that you can get 1800Mbps on 1Gig<br>
bidirectionaly but i<br>
am able to get<br>
outbound:560mbps<br>
inbound:450mbps<br>
<br>
<br>
wat can be done to make it to 1800 Mbps? I will be very thankful<br>
if you help me out man.<br>
<br>
<br>
You will need to see first what the bottleneck happens to be.<br>
Attached to this message is some boilerplate I have worked-up that<br>
may help.<br>
<br>
Also, what sort of I/O connection does your dual-port chip have on<br>
this board? PCIe? PCI-X? Speeds and feeds?<br>
<br>
rick jones<br>
lets keep this discussion on netperf-talk for the benefit of all.<br>
<br>
<br>
Thanks<br>
<br>
<br>
<br>
<br>
<br>
On Thu, Jul 9, 2009 at 10:30 PM, Rick Jones <<a href="mailto:rick.jones2@hp.com" target="_blank">rick.jones2@hp.com</a><br></div></div>
<mailto:<a href="mailto:rick.jones2@hp.com" target="_blank">rick.jones2@hp.com</a>> <mailto:<a href="mailto:rick.jones2@hp.com" target="_blank">rick.jones2@hp.com</a><div><div></div><div class="h5"><br>
<mailto:<a href="mailto:rick.jones2@hp.com" target="_blank">rick.jones2@hp.com</a>>>> wrote:<br>
<br>
Ankit Goyal wrote:<br>
<br>
Thanks a ton guys!!<br>
If possible could you tell me that how much max<br>
bidirectional<br>
throughput I can get on 1Gbps ethernet connection?Can I<br>
get more<br>
than 1 Gig by changing the drivers and kernel?<br>
I know its a very relative question but i will be glad if<br>
u help<br>
me out!<br>
<br>
<br>
In theory you should be able to see O(1800) megabits/s.<br>
Certainly<br>
that would be my expectation in this day and age:<br>
<br>
<br>
s7:/home/raj/netperf2_trunk# netperf -H sbs133b1.west. -t<br>
TCP_RR -f<br>
m -- -r 64K -s 256K -S 256K -m 32K -b 8<br>
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0<br>
AF_INET to<br>
sbs133b1.west (10.208.1.20) port 0 AF_INET : first burst 8<br>
Local /Remote<br>
Socket Size Request Resp. Elapsed<br>
Send Recv Size Size Time Throughput<br>
bytes Bytes bytes bytes secs. 10^6bits/sec<br>
<br>
262142 262142 65536 65536 10.00 1837.52<br>
524288 524288<br>
<br>
whether one will always get that with the paired<br>
TCP_STREAM/TCP_MAERTS test is an open question:<br>
<br>
s7:/home/raj/netperf2_trunk# for i in 1; do netperf -t<br>
TCP_STREAM -l<br>
60 -H sbs133b1.west -P 0 & netperf -t TCP_MAERTS -l 60 -H<br>
sbs133b1.west -P 0 & done<br>
[1] 14619<br>
[2] 14620<br>
s7:/home/raj/netperf2_trunk#<br>
87380 16384 16384 60.02 713.28<br>
87380 16384 16384 60.01 874.86<br>
<br>
s7:/home/raj/netperf2_trunk# for i in 1; do netperf -t<br>
TCP_STREAM -l<br>
120 -H sbs133b1.west -P 0 & netperf -t TCP_MAERTS -l 120 -H<br>
sbs133b1.west -P 0 & done<br>
[1] 14621<br>
[2] 14622<br>
s7:/home/raj/netperf2_trunk#<br>
87380 16384 16384 120.03 626.89<br>
87380 16384 16384 120.01 895.40<br>
<br>
(FWIW, one of the systems involved there is several (4 or 5?)<br>
years<br>
old now)<br>
<br>
<br>
<br>
<br>
Some of my checklist items when presented with assertions of poor<br>
network performance, in no particular order:<br>
<br>
*) Is *any one* CPU on either end of the transfer at or close to 100%<br>
utilization? A given TCP connection cannot really take advantage<br>
of more than the services of a single core in the system, so<br>
average CPU utilization being low does not a priori mean things are<br>
OK.<br>
<br>
*) Are there TCP retransmissions being registered in netstat<br>
statistics on the sending system? Take a snapshot of netstat -s -t<br>
from just before the transfer, and one from just after and run it<br>
through beforeafter from<br>
<a href="ftp://ftp.cup.hp.com/dist/networking/tools" target="_blank">ftp://ftp.cup.hp.com/dist/networking/tools</a>:<br>
<br>
netstat -s -t > before<br>
transfer or wait 60 or so seconds if the transfer was already going<br>
netstat -s -t > after<br>
beforeafter before after > delta<br>
<br>
*) Are there packet drops registered in ethtool -S statistics on<br>
either side of the transfer? Take snapshots in a matter similar to<br>
that with netstat.<br>
<br>
*) Are there packet drops registered in the stats for the switch(es)<br>
being traversed by the transfer? These would be retrieved via<br>
switch-specific means.<br>
<br>
*) What is the latency between the two end points. Install netperf on<br>
both sides, start netserver on one side and on the other side run:<br>
<br>
netperf -t TCP_RR -l 30 -H <remote><br>
<br>
and invert the transaction/s rate to get the RTT latency. There<br>
are caveats involving NIC interrupt coalescing settings defaulting<br>
in favor of throughput/CPU util over latency:<br>
<br>
<a href="ftp://ftp.cup.hp.com/dist/networking/briefs/nic_latency_vs_tput.txt" target="_blank">ftp://ftp.cup.hp.com/dist/networking/briefs/nic_latency_vs_tput.txt</a><br>
<br>
but when the connections are over a WAN latency is important and<br>
may not be clouded as much by NIC settings.<br>
<br>
This all leads into:<br>
<br>
*) What is the *effective* TCP (or other) window size for the<br>
connection. One limit to the performance of a TCP bulk transfer<br>
is:<br>
<br>
Tput <= W(eff)/RTT<br>
<br>
The effective window size will be the lesser of:<br>
<br>
a) the classic TCP window advertised by the receiver (the value in<br>
the TCP header's window field shifted by the window scaling<br>
factor exchanged during connection establishment (why one wants<br>
to get traces including the connection establishment...)<br>
<br>
this will depend on whether/what the receiving application has<br>
requested via a setsockopt(SO_RCVBUF) call and the sysctl limits<br>
set in the OS. If the application does not call<br>
setsockopt(SO_RCVBUF) then the Linux stack will "autotune" the<br>
advertised window based on other sysctl limits in the OS.<br>
<br>
b) the computed congestion window on the sender - this will be<br>
affected by the packet loss rate over the connection, hence the<br>
interest in the netstat and ethtool stats.<br>
<br>
c) the quantity of data to which the sending TCP can maintain a<br>
reference while waiting for it to be ACKnowledged by the<br>
receiver - this will be akin to the classic TCP window case<br>
above, but on the sending side, and concerning<br>
setsockopt(SO_SNDBUF) and sysctl settings.<br>
<br>
d) the quantity of data the sending application is willing/able to<br>
send at any one time before waiting for some sort of<br>
application-level acknowledgement. FTP and rcp will just blast<br>
all the data of the file into the socket as fast as the socket<br>
will take it. scp has some application-layer "windowing" which<br>
may cause it to put less data out onto the connection than TCP<br>
might otherwise have permitted. NFS has the maximum number of<br>
outstanding requests it will allow at one time acting as a<br>
defacto "window" etc etc etc<br>
<br>
<br>
</div></div></blockquote>
<br>
</blockquote></div><br>