I'm troubleshooting a low throughput replication transfer on a OC12 circuit with 36 msec RTT between two sites. The TCP handshake in the trace shows the advertised scaling for the recipient side of the data to be 1MB WS-5. Based on the calculation, the throughput should be around 220Mbps but it's only 14.6Mbps over an OC12 that's barely being utilized. Based on the 14Mbps throughput, I estimated the Window size is more 64KB even though the recipient is advertising 1MB. Retransmissions and network are not an issues so I don't know the reason for the low throughput. The appliances are running Solaris and this is a ZFS transfer. The TCP settings for the window parameters are set from the the vendor with a 1MB buffer and cannot be changed. Not sure what else I can look for in the trace. asked 24 May '12, 16:45  ws2006 |
2 Answers:
From just looking at the bare numbers, I'd expect that the client advertises 64k as his recieve window, because the Bandwidth-Delay-Product perfectly points to around 65.700something with your given 36msec and 14.6Mbps so few possibilities to check first:
answered 25 May '12, 01:06  Landi |
in general, packet size (MTU and thus MSS) does have an impact on max. TCP throughput over WAN links (search for Mathis).
However in your case that's not relevant.
So, there must be some other effect in place while the MTU size was set to 1500, like packets being dropped due to different MTU sizes and fragmentation. Look for "icmp fragmentation needed" messages in the orignal capture and look at the max. packet size you see in the capture file. Are they remarkably smaller than the MTU size (~1500 or whatever was set)? Regards answered 31 May '12, 18:29  Kurt Knochner ♦ edited 31 May '12, 19:36 The initial replication was one stream (share) with a 14-16Mbps throughput. 8 TCP share sessions were kicked off with a total of 120-128Mbps with still plenty of bandwidth on an OC12. The MTU of 4000 changed was made on the network and with the new 8 shares running and the OC12 was fully utilized. i was able to capture the initial share and it showed both sides with WS announcement as before with 1MB window size and the throughput peaked to over 220Mbps. The rest of the seven shares kicked off thereafter and I assume each tcp session probably ws to 1MB. Since it was an OC12(622Mbps), it could possibly went over the OC12 if there were more bandwidth. I will ask them to start with 3 shares to see if that will fill up the oc12. i calculated the throughput as window size in bits / rtt. 1,024,000 * 8 = 819,200 / .036 = 227Mbps. (01 Jun '12, 03:27) ws2006 @ws2006 Please don't post comments as "answers" that's confusing for other users of the site. I've converted your "answer" to a comment. (01 Jun '12, 03:44) grahamb ♦ 220Mbps, is that the aggregated throughput measured as total of the 8 TCP connections? If so, what is the throughput of one connection with 4k MTU? (01 Jun '12, 03:55) Kurt Knochner ♦ the 220Mbps was the peak i saw for in the initial TCP connection (share) for a few seconds. The rest of the sessions kicked off at a few minutes later interval and don't have the trace for those so don't know each of the sessions throughput. Since the the OC12 was maxed around 600Mbps for a few hours, 600Mbps / 8 shares = 75Mbps for each share. 75,000,000 * .036 = 2,700,000 / 8 = 337,500 bytes window size for each session? For the trace, the TCP Len is 1368 and Wireshark reporting packet size limited during capture: JXTA truncated. (01 Jun '12, 05:44) ws2006 |
can you please randomize and truncate the capture file and then upload it somewhere?
It's hard to help without the real numbers ....
i will try to get the capture uploaded. I do have a question on the TCP handshake. Sender1 will be sending the bulk of the data to the receiver1. Sender1 advertised window size is 128KB and the receiver1 advertised its window size of 1MB. After the handshake, receiver1 show 1MB and sender1 show 128KB calculated window size in the trace. Since receiver1 is the receiving the bulk of the data and advertising a 1MB window, the sender should be able to send about 220Mbps but can only send 14-17Mbps of data. My assumption is the sender is only using a 64KB window size based on 65536 * 8 = 524288 bits / .037 RTT = 17Mbps. I would expect by the receiver advertising of 1MB the sender will use this window size. There are not wan accelerators or traffic shaping devices on the network other than firewall.
maybe the firewall adds delay for some reason. Probably we will see that in the capture file.
A network change was made to the MTU size for both ends to 4000. Each server at each end also modified the NIC to have a 4000 MTU size. The advertised window size is still 1MB for each server. The WAN is 622Mbps circuit with .036 second RTT. They are now able to get an actual throughput of 220Mbps. I calculated as 1,024,000 * 8 bits = 8192000 / .036 = 227Mbps. I still not sure how the MTU change made the difference and how would I see this in the trace. Any advice? Thanks.