I am trying to troubleshoot some issues on a Linux Apache web server (very slow response several times during the day). For testing i am trying to load a page on my laptop (192.168.249.2) from the web server (172.18.26.41).
I used wireshark and noticed some TCP retransmissions and TCP DUP ACKs, see below, I ran the trace on both ends and saw the same results so no packets are going missing. I cant see anything out of sequence here and i don't understand why 26.41(web) is sending a TCP retransmission (22461) if its already received an ACK in packet 21453.
It only does this when there is a heavy load on the server, could this be an application (Apache) issue where port 80 becomes unavailable, could wireshark have seen packet 21453 come in but if apache is busy and port 80 becomes unavailable 26.41 issues a re transmission??
There doesn't appear to be any congestion on the network, we have 100mb link into the MPLS and a 1gb at the other end and the highest utilisation is approx 30%.
Any help would be helpful.
asked 12 Jun ‘14, 03:18
edited 12 Jun ‘14, 03:20
After doing a googling found that it could be causing because of TCP_DEFER_ACCEPT option,Please check similar behaviour in this link, http://webcache.googleusercontent.com/search?q=cache:zoA_A0cao28J:https://labs.ripe.net/Members/gih/the-curious-case-of-the-crooked-tcp-handshake+&cd=9&hl=en&ct=clnk&gl=in
answered 14 Jun '14, 00:35
Based on the answer of @kishan pandey (TCP_DEFER_ACCEPT), here is a new attempt to explain what could have happened:
TCP_DEFER_ACCEPT makes the server wait for a data packet from the client, which never arrives, so the SYN-ACK is sent again.
Now, the interesting question is: Why is there no data packet (HTTP GET, POST, HEAD) from the client?
A possible answer: Because it's not the client who ACKs the SYN-ACK, but 'something' between the client and the server. And now we should have a look at your Fortinet Firewall (concluded from the MAC address used in the capture file).
Maybe the Fortinet acts as a TCP or SYN proxy, because there is a content scanning feature enabled on that firewall and that's causing problems.
Maybe the client sends a SYN and the firewall forwards that SYN to the server. The server answers with a SYN-ACK (wrong check sum in the capture file), which gets answered by the firewall (SYN proxy, TCP proxy) with an ACK. But then the firewall does not forward the SYN-ACK to the client and also drops any further SYN (re-transmit) from the client, for whatever reason (Hint: the SYN-ACK from the server shows a wrong check sum in the capture file - which could be true or caused by TCP offloading on the server).
As the client has no chance to send the data packet (never seen the SYN-ACK), the server will resend its SYN-ACK due to the TCP_DEFER_ACCEPT option.
So, the key to analyze this problem is a capture file taken at the client and the server in parallel, to check who is doing what.
@navs_123456: Can you please post both capture files (client and server), so we can check my new theory :-)
answered 15 Jun ‘14, 03:24
Kurt Knochner ♦
edited 15 Jun ‘14, 03:32
Looks like the ACK for the SYN-ACK does not get through to the server, so the SYN-ACK is sent again and again.
You have created the capture file on (or near) the client. What do you see in the capture file on (or near) the server at the same time?
answered 12 Jun '14, 05:19
Kurt Knochner ♦
edited 12 Jun '14, 07:10
Maybe the TCP or IP checksumm is incorrect. The trace file is gone again so I couldn't check myself
Ok, found the trace by removing the '.' from the URL and the checksums are correct.
answered 12 Jun '14, 13:13
edited 13 Jun '14, 12:01