Hi I am trying to troubleshoot speed issues for a customer connecting to a Web Site from abroad. We have carried out testing by asking carrying out downloads of files from our Web site & capturing traffic during this. We can see that once the download commences, after a short amount of packets, the TCP Window size on the receiving host drops to zero. After a short while this resumes & then does not drop below 40000. On this machine the Windows Scaling option is set to 3 & the TCP Window size set to 65535. Even with the scaling option set the TCP Window size never goes above 65535 in the packets (which I am told is normal). I am surprised that, with the scaling set to 3, the TCP Window size still reduces to zero. IS this normal behaviour? I am guessing that the TCP Window reducing to zero is because of some resource limitation on the Receiving host...? Any assistance would be greatly appreciated. Thanks Ian asked 16 Feb '11, 05:07  ipittam |
3 Answers:
TCP window dropping to Zero means that the receiving host has trouble processing incoming data fast enough. Even with window scaling it is something that can happen - the larger the window the longer it takes to drop to zero. If the receiving station is too slow to process data and it has to receive more data than the window size is in size it will go down to zero. You can take a look at the "bytes in flight" value in the sender's tcp header to see how much data has not yet been acknowledged - it ususally shows pretty high numbers if the receiver is too slow to process the incoming flood of packets. Zero windows very often indicate an overloaded receiving station (either OS or slow application), and is a good indicator that "it is NOT the network" :-) Exception are Reset-Packet - they usually have a window size of 0 and are not an indication of an overworked station. answered 16 Feb '11, 05:31  Jasper ♦♦ edited 16 Feb '11, 05:32 |
When you look at that station communicating - how do you know the Window Scale is in use? Do you see a "scaled" value in the host's TCP headers? Here's the deal and it's all over the place these days. - your client can be set to use Window Scaling (TCP options in handshake) - if the server doesn't support it, neither side will use it (look at server handshake response) - if an intermediary device (such as a "smart firewall" - ugh) doesn't like/support that option for the path, it might strip off your client's TCP option (thinking Cisco ASA here). - if Window Scaling is on and truly in use and your client still hits Window Zero conditions, then what the heck is the application running for that communication? Example: IE being dog slow picking data up out of receive buffer causes Window Zero condition on large file download - update IE or get a better browser. BTW, the Window Size field can only go to 65,535 as it is a 2-byte field. If Window Scaling is truly in use and you have set the TCP Preferences to show Relative Sequence Numbering and Window Scaling (the default), then you should see a scaled factor after the Window Size field in the communications after the first two packets of the handshake. answered 16 Feb '11, 11:17  lchappell ♦ Thanks for your comments - very useful! We have convinced our Service Provider to enable TCP windows Scaling & Selective Acks. We now see that the these options are mutually enabled it the SYN / SYN-ACK packets. A further observation we have is that the when downloading a file from a Web Server behind the Cisco ACE device via a SSL connection (HTTPS URL) we can see that the Cisco is struggling with flow & reducing the TCP Window & also the Server advertising that the TCP Window is full. Thanks Ian (18 Feb '11, 09:27) ipittam Am I right in saying that this could be too do with processing delays on the ACE device due to SSL processing? Could it also be due to a downstream device throttling the connection & this being passed down the stream to the ACE & eventually to the sending Web Server? (Web Server > Cisco ACE > Intermediate router > ....Client) Thanks Ian (18 Feb '11, 09:27) ipittam SSL processing might be a reason why the window drops to zero since it adds further encryption load on the devices apart from just transfering data. If you still have the zero window problem you should track down the origin of which device advertises that window and performance tune it or (worst case) replace it with a more powerful device. (19 Feb '11, 11:19) Jasper ♦♦ Ok... not to cast stones or anything.... but... "the Cisco is struggling with flow & reducing the TCP Window"... that makes me gag. Aren't these guys the gurus of infrastructure and data flow? I'm gonna just say it outright... they stink if they are adjusting the Window Size value - do we have any Cisco folks hanging around here? I've seen Cisco products kill SACK in the handshake too... can you take a trace at both ends of a TCP handshake and see what they are doing to the TCP options fields? ... breathing deeply now... going to my happy place... <g> (21 Feb '11, 02:13) lchappell ♦ I can see you being able to selectively disable SACK in ACE. Unless the ACE box is being hammered I don't see it being the choke point for the SSL communications, those boxes have ASICs dedicated to crypto. (22 Feb '11, 06:26) GeonJay If there IS an ACE module please really be SURE that both ends of the communication really have identical TCP headers + options... We also had issues with cisco devices cutting the options field during transfer which only came out doing a multi-point trace. So like Laura said keep that in mind and because I nearly vomited at that point at a recent project I could not not comment this here :) (28 Feb '11, 06:31) Landi showing 5 of 6 show 1 more comments |
Usually the window scaling should be set to a value high enough to continuously transfer packets without having to stop and wait for an ACK to arrive. The bytes in flight can not exceed the effective window size. If window scaling is enabled you'll see that both station negotiate the window scale factor in their SYN packets, which is then used to calculate the effective window size as "2 to the power of <scale factor=""> multiplied by the 'normal' 16bit window size". The latest development builds of Wireshark show both values, "normal" window size and scaled window size. I have seen intermediate devices change the window size, resulting in a very ineffective communication (usually packet shaping or prioritizing devices, layer3 devices usually don't do this). You can check that by comparing window size and scale factors when capturing both at client and server: both should show the same values - if not, an intermediate device fumbles around with your packets. answered 16 Feb '11, 08:33 Jasper ♦♦ Thanks Jasper So is the fact that we see different communication conversations at each end meaning that the intermediate networks devices are altering the flow. Am I right in presuming that this flow control would be different for each intermediate hop? Thanks again! Ian (16 Feb '11, 09:35) ipittam What kind of differences do you see? Different window sizes and/or scale factors for the exact same packets? Are there any devices doing NAT? I'm not sure what you mean with "different flow control for intermediate hops". Usually TCP flows are not modified by network devices except when NATting, or if a traffic shaper/prioritizing device is used. (16 Feb '11, 15:33) Jasper ♦♦ |
Hi Jasper,
Thanks for your response.
I was looking at the "bytes in flight" value in the packets (we have a packet capture on the client side for a download). I can see that bytes in flight never goes beyond 2736 (ie 2 packets) at a given time even though the client has been showing a win size of 65535.
Should the server be not sending more data (and hence show more bytes under bytes in flight) and thus make the download faster?
Are we missing something? Thanks - Ian
For a capture at the client side that is pretty normal as every other packet is acknowledged (thus resulting in 2 packets being "in flight", meaning "not acknowledged"). On the server side you should see higher numbers of bytes in flight though, at least if client and server are not very close to each other.
Regardless of the bytes in flight the important fact is still the Zero Window syndrome - this should not happen in the middle of a transfer because it means the receiver is too slow to process data.
Thanks Jasper
We have looked at the Packet capture from the Server & it is quite different. The bytes in flight reaches up to almost 65535 & then ACKs are received.
If the TCP Windows Scaling setting is enabled, should the bytes in flight be seen to increase above this amount?
Also, am I right in saying that the Window Size setting may be changed by intermediate network devices, as the packets are buffered by these devices in transit?
Thanks Ian