Slow Transfer, Dup Ack, Packets out of order - Only one direction

Question

I have a client that I am seeing very weird behavior on their network. When I transfer data TO the site in Phoenix across the MPLS everything is great, full bandwidth beautiful packet captures. When I transfer data FROM the Phoenix site across the MPLS I get 1/3 bandwidth and ugly packet captures, missed packets, 100s of DUP ACKS and packets out of order.

I don't see anything wrong with layer 1, 2, or 3, but I have seen that the window size tanks pretty early on the transfers FROM Phoenix. I've got packet captures, I'll get them posted here shortly.

Answer 1

It could be either a MPLS routing issue or perhaps a load balancer issue.

Perform tracert and tracetcp (including port number) and compare. Do this on both sides to see if there are any route differences.

Answer 2

I've sent a traceroute over to the ISP and they confirmed that it is going over the correct path, even though they do show differently. I paid for a subscription to cloudshark so that I could upload these.

I did a port span on the switch because I wasn't sure the test machine had enough CPU to capture and process the iPerf test.

When the client is in San Diego sending to the iperf server in Scottsdale everything is beautiful. https://www.cloudshark.org/captures/6c93d681ff72

When the client is in Scottsdale sending to San Diego, it all goes to crap. https://www.cloudshark.org/captures/93d5799d876b

Just for posterity here are the paths that I show across the MPLS as well.

San Diego Network to Scottsdale Network 1 1 ms <1 ms <1 ms 192.168.1.2 2 8 ms 7 ms 8 ms 10.67.1.61 3 22 ms 19 ms 20 ms 82.197.9.72.static.ip.tnltd.net [72.9.197.82] 4 20 ms 20 ms 19 ms 10.251.27.216.static.ip.tnltd.net [216.27.251.10] 5 19 ms 19 ms 19 ms server.scottsdale [192.168.0.19]

Scottsdale to San Diego Network 1 1 ms 1 ms 1 ms 192.168.0.1 2 <1 ms <1 ms <1 ms 192.168.0.2 3 2 ms 2 ms 2 ms 10.254.4.201 4 * * * Request timed out. 5 * * * Request timed out. 6 12 ms 11 ms 11 ms 10.67.1.61 7 19 ms 19 ms 19 ms 10.67.1.62 8 19 ms 20 ms 19 ms server.san.diego [192.168.1.6]