Need advice with trace file

Question

This issue is with the client timing out when accessing a particular web site going through a proxy appliance. The client can access all other Internet web sites with no issue with exception of this url that had previously worked fine. A capture was done on the bluecoat proxy and it shows the http 200 ok response being sent to the client. For the client side, the firewall capture does not show the reply reaching the firewall. No visibility along the paths after the bluecoat so don’t know where it could have been loss. It is odd that only this web site is affected and all other web sites are working. At this time, all firewall, bluecoat and routing issues have been ruled out and root cause is unknown at this time.

The trace file is can be found at http://cloudshark.org/captures/4a3b7c2a3230

Request: Client-?Client site FW<---?FW1----?ISP--?Data Center Core/Proxy--?Internet

Reply: Internet-?Data Center Core/Proxy–?ISP-?FW1–?Client site FW–?Client client FW NAT IP: 192.168.1.1 Proxy:192.168.1.2

FW1 capture is shown below.

13:22:17.   0x0800 62: 192.168.1.1.50863 > 192.168.1.2.80: S [tcp sum ok] 2755830644:2755830644(0) win 8192 <mss 1260,nop,nop,sackOK> (DF) (ttl 126, id 6888) : SYN
13:22:17.   0x0800 62: 192.168.1.2.80 > 192.168.1.1.50863: S [tcp sum ok] 2416274753:2416274753(0) ack 2755830645 win 65535 <mss 1350,sackOK,eol> (ttl 60, id 37726) : SYN, ACK
13:22:17.   0x0800 60: 192.168.1.1.50863 > 192.168.1.2.80: . [tcp sum ok] 2755830645:2755830645(0) ack 2416274754 win 65520 (DF) (ttl 126, id 6889) : ACK
13:22:17.   0x0800 365: 192.168.1.1.50863 > 192.168.1.2.80: P 2755830645:2755830956(311) ack 2416274754 win 65520 (DF) (ttl 126, id 6890) : GET HTTP
13:22:18.   ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? id 38997 is missing (id 38997 is in the trace packet capture from the bluecoat).
13:22:18   0x0800 392: 192.168.1.2.80 > 192.168.1.1.50863: P 2416276014:2416276352(338) ack 2755830956 win 65535 [tos 0x60]  (ttl 60, id 38998) : TCP segment of a reassembled PDU
13:22:18   0x0800 66: 192.168.1.1.50863 > 192.168.1.2.80: P [tcp sum ok] 2755830956:2755830956(0) ack 2416274754 win 65520 <nop,nop,sack sack 1 {2416276014:2416276352} > (ttl 255, id 46285) : TCP Dup ACK4#1
13:22:18.   0x0800 365: 192.168.1.1.50863 > 192.168.1.2.80: P 2755830645:2755830956(311) ack 2416274754 win 65520 (DF) (ttl 126, id 6897) : TCP Retransmission

Answer 1

The client sends an MSS of 1260 to the bluecoat and the bluecoat does indeed follow up with sending packets with 1260-byte segments. But from your tracefile on Cloudshark one can see that all frames of length 1314 (1260 bytes payload) don't get to the client. This can be seen by looking at frame 9 in which the client acknowledges receiving frame 6 but not frame 5 (look at the SACK edges).

To me this looks like an MTU issue. There might be a link in the network between the bluecoat and the client that is not capable of transporting 1314 byte frames. Normally the frames would be fragmented, but either fragmenting is disabled or a fw drops the frames which are fragmented.

You might want to use ping with different packet sizes and the DF bit set to determine the actual smallest MTU of all the links between the client and the bluecoat.