tcp mss inconsistency in different capture points

Question

Analysis process, the client is found in the TCP connection is established, the MSS value is not the same in different capture points. client side packet: http://www.cloudshark.org/captures/0c7d36890433 server side packet: http://www.cloudshark.org/captures/1d460ea5291d

Help to explain

thanks.

Answer 1

3

There has to be a device sitting in the middle has changed the MSS value, since not only the MSS is changed but also the initial sequence number. So this looks like a transparent proxy of some sort, which you may be able to verify by sending an HTTP TRACE request. If the server responds to it it will echo back the request parameters the server got and display them in your browser (guessing that it is a Web server from port 443 being used).

answered 19 Apr '13, 10:02

Jasper ♦♦
23.8k●5●51●284
accept rate: 18%

In the middle of the client and the server in a IPSEC VPN tunnel,but not have a transparent proxy.

(19 Apr '13, 10:11) mengsunny

This environment will cause the change of MSS value

(19 Apr '13, 10:12) mengsunny

edited: [IPSEC tunnels (basically routers with next hop encryption) usually do not modify layer 4 information, but it may be still possible] not correct, see comment of @SYN-bit - . You need to determine all devices in the path and what they're configured to do with layer 4 information.

(19 Apr '13, 11:08) Jasper ♦♦

@Jasper, it is best practice in IPsec tunneling to adjust the MSS of TCP sessions to prevent fragmentation at the IP layer due to the encapsulation.

Cisco even takes this further and by default changes the MSS of TCP sessions to 1380 on the FWSM and ACE blades.

See for instance:

(19 Apr '13, 12:18) SYN-bit ♦♦

Makes sense, I guess, I just haven't seen many traces like that before :) I was just wondering why a device would also create a new inital sequence number - and it led me to the assumption that there is some kind of proxying involved.

(19 Apr '13, 15:03) Jasper ♦♦

That's also a cisco FW feature for the ol'days when systems still used 0 as ISN. Cisco firewalls randomize the ISN for the internal hosts... And guess what, that's why they could not handle SACK in the past (or maybe they still don't?). You can turn the ISN randomization of though :-)

(19 Apr '13, 16:19) SYN-bit ♦♦

Thank you very much for your help, I will continue to eliminate do operation for 4 layer equipment possible path.

(19 Apr '13, 20:17) mengsunny

Your answer has been converted to a comment as that's how this site works. Please read the FAQ for more information.

If an answer has solved your issue, please accept the answer for the benefit of other users by clicking the checkmark icon next to the answer. Please read the FAQ for more information.

(20 Apr '13, 05:23) grahamb ♦

Hm, interesting to see that the windows client is NOT honoring the reduced MSS of 1380 but sends a tcp.len==1477 segment in frame.number 13. This packet also exceeds the client's MTU size by 17 bytes... The re-transmission in #15 after 314 ms is then using the smaller - learned by PMTUD? - MTU size and this packet goes through. Is this 'segmentation offload' or is this normal windows behaviour?

(21 Apr '13, 06:07) mrEEde2

It does honor it as far as I can tell. Keep in mind that the trace taken on the client side was taken on the client. That means that you deal with "oversized" segments because of segmentation offloading. Check the server side to see that the MSS of 1380 is never violated.

(21 Apr '13, 06:14) Jasper ♦♦

The large segment never makes it to the server. So if TSO is enabled, the ethernet card does not look at the 'learned' MSS but sends out too large segments requiring a retransmission with the new (PMTUD learned) MTU size. So we'll always have to wait for the retransmission timer to pop - not good for performance.

(21 Apr '13, 06:26) mrEEde2

I agree with @Jasper that TSO is indeed enabled, as you can see that the 1477 bytes of TCP data from (clienttrace) frame 13 is split into 1380 bytes (missing in the server trace) and 97 bytes in frame 25 (servertrace).

The real problem is why the frame with 1380 bytes of TCP data did not get through, as this is conform the advertised MSS. Please see my separate answer for a possible cause.

(21 Apr '13, 06:37) SYN-bit ♦♦

Right, I was only checking tcp.len on the server side, and ignore the packet loss - my bad. Looks indeed like the segmentation offloading of the client doesn't consider MSS until it is forced to retransmit. Or it does, as SYN-bit mentioned - I think I need to clear my head and look through this again some time. These sequence number replacement mechanisms give me a headache :)

(21 Apr '13, 06:40) Jasper ♦♦

@mengsunny: What kind (brand, modell) of VPN device is involved?

(21 Apr '13, 08:04) Kurt Knochner ♦

to answer my own question :-)

Based on the ICMP "fragmentation needed" message (frame #57), I conclude, that the sending device (21.238.5.11) used a TTL of 255. As Cisco routers (and possibly Cisco ASA as well) are using that TTL value, I assume it's some Cisco IPSEC device and then the explanation of @SYN-bit sounds pretty reasonable.

(21 Apr '13, 08:17) Kurt Knochner ♦

showing 5 of 15 show 10 more comments

Answer 2

The traces look a lot like traces I have made before for a customer where things did not work properly. In the end it was the IPsec configuration on the Cisco routers that caused the issue. You could check whether the crypto map is applied to the tunnel interface instead (or as well as) the physical interface. In my case the solution was to have the crypto map on the physical interface only.

Please look for "ICMP Fragmentation needed but DF bit set" messages on the client side which suggest an MTU of 1433 (which corresponds to an MSS of 1379 instead of 1380). That was what we were facing back then and we created a Cisco case.

This was the response from Cisco-TAC in my case:

So here's what we are seeing on the show crypto ipsec sa: interface: Tunnel0 Crypto map tag: GENVPN, local addr x.x.x.141 ... local crypto endpt.: x.x.x.141, remote crypto endpt.: y.y.y.206 path mtu 1476, ip mtu 1476, ip mtu idb Tunnel0 current outbound spi: 0x0(0) This is incorrect. The ipsec SADB should only be associated with the physical interface, and not the Tunnel interface. Because of that, we see that even for the g0/1 interface, the MTU is being computed from the Tunnel interface mtu: interface: GigabitEthernet0/1 Crypto map tag: GENVPN, local addr x.x.x.141 … local crypto endpt.: x.x.x.141, remote crypto endpt.: y.y.y.206 path mtu 1476, ip mtu 1476, ip mtu idb Tunnel0 current outbound spi: 0x0(0)

This is one of the "unexpected" results of having the crypto-map applied to the Tunnel interface. Because of that all the MTU computations are being done with basis the MTU of the GRE tunnel, wich is 24 less than the physical interface. I could reproduce the issue in the lab, and by provisioning the crypto-map on the physical interface, before provisionning it on the Tunnel, the MTU will be derived from the physical interface, until next reload. This is most probably what happened in your case, given the crash that we have identified. Therefore the solution is quite simple, simply remove the crypto map statement from the tunnel interface configuration. You might need to clear all the crypto tunnels or reload the box, if the removal is not enough. In the lab removing the crypto map was enough, but I did not have redundancy configured, so please keep that in mind. Please keep in mind that this is an unsupported configuration as it is right now.