Some of my checklist items when presented with assertions of poor
network performance, in no particular order:

*) Is *any one* CPU on either end of the transfer at or close to 100%
   utilization?  A given TCP connection cannot really take advantage
   of more than the services of a single core in the system, so
   average CPU utilization being low does not a priori mean things are
   OK.

*) Are there TCP retransmissions being registered in netstat
   statistics on the sending system?  Take a snapshot of netstat -s -t
   from just before the transfer, and one from just after and run it
   through beforeafter from
   ftp://ftp.cup.hp.com/dist/networking/tools:

   netstat -s -t > before
   transfer or wait 60 or so seconds if the transfer was already going
   netstat -s -t > after
   beforeafter before after > delta

*) Are there packet drops registered in ethtool -S statistics on
   either side of the transfer?  Take snapshots in a matter similar to
   that with netstat.

*) Are there packet drops registered in the stats for the switch(es)
   being traversed by the transfer?  These would be retrieved via
   switch-specific means.

*) What is the latency between the two end points.  Install netperf on
   both sides, start netserver on one side and on the other side run:

   netperf -t TCP_RR -l 30 -H <remote>

   and invert the transaction/s rate to get the RTT latency.  There
   are caveats involving NIC interrupt coalescing settings defaulting
   in favor of throughput/CPU util over latency:

   ftp://ftp.cup.hp.com/dist/networking/briefs/nic_latency_vs_tput.txt

   but when the connections are over a WAN latency is important and
   may not be clouded as much by NIC settings.

   This all leads into:

*) What is the *effective* TCP (or other) window size for the
   connection.  One limit to the performance of a TCP bulk transfer
   is:

   Tput <= W(eff)/RTT

   The effective window size will be the lesser of:

   a) the classic TCP window advertised by the receiver (the value in
      the TCP header's window field shifted by the window scaling
      factor exchanged during connection establishment (why one wants
      to get traces including the connection establishment...)
   
      this will depend on whether/what the receiving application has
      requested via a setsockopt(SO_RCVBUF) call and the sysctl limits
      set in the OS.  If the application does not call
      setsockopt(SO_RCVBUF) then the Linux stack will "autotune" the
      advertised window based on other sysctl limits in the OS.
 
   b) the computed congestion window on the sender - this will be
      affected by the packet loss rate over the connection, hence the
      interest in the netstat and ethtool stats.

   c) the quantity of data to which the sending TCP can maintain a
      reference while waiting for it to be ACKnowledged by the
      receiver - this will be akin to the classic TCP window case
      above, but on the sending side, and concerning
      setsockopt(SO_SNDBUF) and sysctl settings.

   d) the quantity of data the sending application is willing/able to
      send at any one time before waiting for some sort of
      application-level acknowledgement.  FTP and rcp will just blast
      all the data of the file into the socket as fast as the socket
      will take it.  scp has some application-layer "windowing" which
      may cause it to put less data out onto the connection than TCP
      might otherwise have permitted.  NFS has the maximum number of
      outstanding requests it will allow at one time acting as a
      defacto "window" etc etc etc