Hello, Our company has a 5MB IP/VPN MPLS between our Seattle and Portland offices. When we move files between offices we see very slow transfer speeds, along the lines of 150kb ps to 350kb ps. I know we aren't going to move files at 4-5mb per second but this seems awfully slow. I grabbed some packets during a transfer from the Netgear Nas in Seattle 192.168.2.10 to my local desktop in Portland 192.168.0.172 and I am hoping someone could take a look and see if you can see anything that will aid in our troubleshooting. I just started at this company so I don't have any history of this issue other than "it's always been slow". You can find the export from wireshark here: https://drive.google.com/file/d/0B6OdS4hheiP2U3Z3RDVkdWx5Ykk/view?usp=sharing Note that before this transfer I set the MTU size on my local network card and on the NAS to 1400 because I thought maybe it was just a lot of fragmentation. This had no effect on the transfer speeds. Here are the results when I ping the NAS from my computer. Pinging 192.168.2.10 with 32 bytes of data: Reply from 192.168.2.10: bytes=32 time=12ms TTL=60 Reply from 192.168.2.10: bytes=32 time=7ms TTL=60 Reply from 192.168.2.10: bytes=32 time=8ms TTL=60 Reply from 192.168.2.10: bytes=32 time=24ms TTL=60 Ping statistics for 192.168.2.10: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 7ms, Maximum = 24ms, Average = 12ms Thanks in advance for any advice you can offer! Danny asked 03 Mar '17, 12:56  dannyq edited 03 Mar '17, 12:58 |
One Answer:
In my opinion, packet losses are a major component of the cause of the slowness here. While the overall throughput averages around just under 2.5 Mbps, there are periods where the instantaneous rate gets close to the maximum 5 Mbps. When the flow ramps up to this rate, we then see multiple packet losses - with the associated Selective ACKs and retransmissions. The flow rate reduces and then ramps up again. At those times of maximum throughput, we have ramped up to just 6 or 7 packets in flight (approx. 10 KB), then we get more packet losses and repeat the cycle. So @dannyq, you should be hunting for the cause of these packet losses in the path between your client and server. We might lean towards looking at the WAN, but it could be anywhere. The fact that they seem to occur when we have about 10 KB "in flight" might be a clue (a device with not much buffer space?). Or (same symptom stated differently) it might the fact that we lose packets when we get close to the 5 Mbps throughput rate (devices at edge of the WAN can't keep up?). Could your MPLS link be oversubscribed by other traffic to/from other offices? If I could, I'd begin by seeing if other MPLS traffic is suffering packet losses. It could, of course, be any of the other usual suspects (e.g., duplex mismatch as already mentioned elsewhere in this post, overloaded device such as a firewall, etc, etc). Finally perhaps we can say what is NOT the problem here. To save any time spent looking in the wrong areas. Window Sizes and Scaling: The RTT for the SMB file transfer seems to be around 8ms but sometimes up to 11ms. At 5 Mbit/sec and with RTT of 10ms, the Bandwidth Delay Product is just 10 KB. So even if TCP Window Scaling was not used (64 KB window), we should still be easily able to fill the 5 Mbps link. Although we don't see the 3-way handshake for the SMB connection, we do see them for some HTTPS connections between the same client and server. These show a client scale of 8 (x256) and a server scale of 7 (x128). The observed in-flight data from the server never fills the client's receive window. We can therefore say that the "slow" flow does not appear to be related to Window Scaling or Window Sizing. answered 05 Mar '17, 17:48  Philst |
Looks like you have got packet loss. Maybe you have somewhere a duplex mismatch in Seattle or Portland.
Try a continuous ping with 1200 byte payload.
What bandwidth is configured / reserved for your link?
A quick analysis using Statistics -> IO Graph reveals a bandwidth of approx. 2 MBit/sec.