Issue with HTTP

Question

we are facing slowness issue with only one of our website .

It is not opening any webpages.

When running Wireshark from client PC we see intial 3 way handshake is fine , but after that suddenly we are getting destination host unreachable and TCP-Out-of-order.

16  24.621544   192.168.1.10    10.150.0.40 TCP 985 [TCP segment of a reassembled PDU]
17  24.621754   10.150.0.40 192.168.1.10    ICMP    81  Destination unreachable (Fragmentation needed)
18  24.621768   192.168.1.10    10.150.0.40 TCP 1457    [TCP Out-Of-Order] http > 58765 [ACK] Seq=894 Ack=2394 Win=6773 Len=1366

Answer 1

0

The trace snippet indicates that PMTUD Path MTU discovery is occuring. The HTTP server tries to send a tcp payload of 1366 bytes and this obviously doesn't get through.

In the ICMP packet you will see the supported MTU size over the path. Normally, you should then see re-transmission occuring with a reduced ip.len and the web page should finally arrive at (be acknowleged by) the client. If this does not occur here, you could artificially reduce the MTU size at your (httpd) end to 1400 (which I suppose is what is being reported back).

answered 03 Jun '13, 03:35

mrEEde2
336●4●6●14
accept rate: 20%

So You are saying we need to change MTU on HTTP server side firewall ?

makes me think why only one server affecetd by that.

(04 Jun '13, 04:46) m_1607

1

I assume that your firewall is not reducing the client's mss option in the syn packet when this server is the target. For your other servers it is probably adjusted. To prove this you need to capture the 3-way handshake to each server and compare the tcp options. Could you please upload your trace upto packet 18 to cloudshark? It is not clear to me why a tcp payload of 1366 bytes coud expand the ip packet to 1457 bytes...

(04 Jun '13, 11:52) mrEEde2

1   0.000000    00:00:00_00:00:00   00:00:00_00:00:00   Ethernet    188 Ethernet Unknown: Invalid length/type: 0x05ff (1535)
2   22.876067   10.150.0.40 192.168.1.10TCP 77  58765 > http [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=4 SACK_PERM=1
3   22.876086   192.168.1.10     10.150.0.40    TCP 103 http > 58765 [SYN, ACK] Seq=0 Ack=1 Win=4380 Len=0 MSS=1460 WS=1 SACK_PERM=1
4   23.140692   10.150.0.40 192.168.1.10TCP 91  58765 > http [ACK] Seq=1 Ack=1 Win=65700 Len=0
5   23.140788   10.150.0.40 192.168.1.10HTTP    630 GET /Pages/default.aspx HTTP/1.1 
6   23.140852   192.168.1.10    10.150.0.40 TCP 91  http > 58765 [ACK] Seq=1 Ack=540 Win=4919 Len=0
7   23.145275   192.168.1.10     10.150.0.40    HTTP    385 HTTP/1.1 401 Unauthorized 
8   23.422818   10.150.0.40 192.168.1.10HTTP    708 GET /Pages/default.aspx HTTP/1.1 , NTLMSSP_NEGOTIATE
9   23.422862   192.168.1.10     10.150.0.40    TCP 91  http > 58765 [ACK] Seq=295 Ack=1157 Win=5536 Len=0
10  23.425676   192.168.1.10      10.150.0.40   HTTP    690 HTTP/1.1 401 Unauthorized , NTLMSSP_CHALLENGE
11  23.697484   10.150.0.40 192.168.1.10HTTP    1328    GET /Pages/default.aspx HTTP/1.1 , NTLMSSP_AUTH, User: \mandar.kulkarni
12  23.697529   192.168.1.10    10.150.0.40 TCP 91  http > 58765 [ACK] Seq=894 Ack=2394 Win=6773 Len=0
13  24.621214   192.168.1.10     10.150.0.40    TCP 1551    [TCP segment of a reassembled PDU]
17  24.621754   10.150.0.40 192.168.1.10ICMP    81  Destination unreachable (Fragmentation needed)
18  24.621768   192.168.1.10     10.150.0.40    TCP 1457    [TCP Out-Of-Order] http > 58765 [ACK] Seq=894 Ack=2394 Win=6773 Len=1366
19  24.621772   192.168.1.10      10.150.0.40   TCP 185 [TCP Out-Of-Order] [TCP segment of a reassembled PDU]

(05 Jun '13, 22:32) m_1607

please help isolate issue

(06 Jun '13, 03:14) m_1607

Just take a look at the dest. unreachable packet in frame 17, unfold the ICMP Header there you will find the needed MTU inside the ICMP message. Why the sender is sending too big packets is then on your side to find out. Check the MTU settings first

(06 Jun '13, 04:02) Landi

I Need to check MTU settings on client side firewall right ?

Internet Control Message Protocol Type: 3 (Destination unreachable) Code: 4 (Fragmentation needed) Checksum: 0x3b4f [correct] MTU of next hop: 1446

Flags: 0x02 (Don't Fragment) 0... .... = Reserved bit: Not set .1.. .... = Don't fragment: Set .1.. .... = Don't fragment: Set

(06 Jun '13, 04:33) m_1607

When there's a firewall in between maybe -> "MTU of next hop: 1446" seen inside the ICMP packet tells you exactly what the MTU in this case is. The SYN from the client says MSS=1460 so MTU 1500 -> doesn't match the 1446 MTU from the dest. unreachable message, so some device inbetween does improper MSS overwrite.

In any case 1551 Bytes in frame 13 is too big for either MTU -> check why the size is that big and if it really is coming from 192.168.1.10 with that size originally

(06 Jun '13, 05:30) Landi

Thanks Landi for your help.

How can we check where does exactly this MTU issue is there, i mean how can we find IP of that device or location of device.

(06 Jun '13, 06:04) m_1607

Bad answer: Check the path and find the device altering MSS values in the SYN packets and/or adding bytes to the data packet. If you can get a tracefile from 192.168.1.10 I'd start there and compare SYN packet values and frame sizes. If you see badly advertised MSS values there start with routers/firewalls to see which device alters

(06 Jun '13, 06:15) Landi

The incoming SYN from the client still carries the original MSS 1460, which means no device has adjusted the MSS in flight, which should have been done if a VPN Tunnel is involved as per the discussion in http://ask.wireshark.org/questions/20633/tcp-mss-inconsistency-in-different-capture-points It would help to see the original .pcap file...

(06 Jun '13, 09:22) mrEEde2

showing 5 of 10 show 5 more comments

Answer 2

Interesting packets, however, it would be much easier to help you if you uploaded the pcap file somewhere (for example to www.cloudshark.org and assuming there is no sensitive data in the pcap file).

A few things that really stand out from the pasted wireshark output:

You say the capture was made on the client, this can not be true as the delta time between SYN and SYN/ACK is 0,021ms while the delta between the SYN/ACK and the ACK is ~264ms. The trace must have been made on the server.
For 1366 TCP payload there is a 1457 byte packet, which results in 91 bytes of headers. 20 for IP, 20 for TCP (no IP or TCP options seem present as the MSS is 1460 in the initial SYN and SYN/ACK). What protocols are used before IP in your case that make up for the other 51 bytes?
Because of NTLM authentication there are a few round trips lost to authentication (between frame 7 and 8 and between frame 10 and 11).
The server response time is ~924ms (see delta between frame 12 and 13)
The delta between sending the first server response packet and the icmp unreachable packet is 0,540ms. The icmp unreachable packet is coming from the same IP as the client. This means it is not the client address but a NAT address for the client and a link between the natting device (10.150.0.40) and the client is using a lower MTU size (94 bytes smaller).

After frame 19, a normal TCP transfer should occur.

In short:

~260ms delay due to high RTT
~540ms delay due to high RTT and authentication
~920ms delay due to high server response time

Apart from that, all seems normal at the TCP level, but I'm curious about the protocols used below the IP layer.

EDIT: As @mrEEde2 is pointing out, it would be best practice for a VPN device to alter the TCP MSS to prevent the extra step caused by the ICMP fragmentation needed.