Hi, I have a server which have several client connected to it, but lately I'm getting this error quite frequent and it caused me to not receive some data needed for this one client and it happened intermittently. As per screenshot, client send SYN and was replied with SYN,ACK by server then later was replied back with ACK, but for some reason the server retransmit back SYN,ACK and later FIN, ACK closing the connection. Does that means that my server network got issue or otherwise. Thanks
asked 28 Jun '15, 23:46  Furqan |
2 Answers:
Yes it like I thought. The client application has fallen into a deep sleep. And I guess that the Request in FRAME 235 and FRAME 265 could really be the root cause. Here you can see a correct Request: And here you can see a delayed request: answered 30 Jun '15, 22:09  Christian_R Hi great finding, but can you elaborate a little bit on the delayed request things, I don't quite understand on it. (30 Jun '15, 23:28) Furqan 1 The reason for the session is to transmit the POST Request which can be seen in FRAME 323. Also you can see in the primary picture that the whole session inclusive Request has a duration of 13ms. Which seems to be fine. But in the second picture the session has a duration of more than 10 Minutes. So what we see is that the session has been initiated at 05:46:48.155. But after the 3WAY Handshake, when normally the POST Request is transmitted nothing happens. So the server starts the discussed "Spurious Retransmission" after ~30 sec. Then ~30 sec after the 3WAY Handshake (05:47:39.491) the server terminates the session by a half close (FIN-Bit). Which is more or less immediately ACKED by the client, so the client is still alive. But the client does not close his half-session, too. Because he still wants to send something. And then after more than 9 Minutes (that is what I meant with delayed) he sends his Request. After that he is glad and closes the session in the same milisecond as the Request, too. The responds this with an RST. In this case the RST has sent, beause the server has closed his socket in the meantime of 9 Minutes. So the question is what causes the client to pause for that long time. Is this the description you were missing or do you need more info? (01 Jul '15, 03:12) Christian_R Thanks for the clear explanation, I guess it's time for the client to check on their server not. Thanks alot!! (02 Jul '15, 01:28) Furqan |
Furqan, Was the first screenshot captured on the server itself? Here are a couple tips to make it much easier for people to assist you with packet captures. 1) If you need to hide IP addresses save the capture in the pcapng format. That will allow you to manually add and save names with the capture file. You can simply right click on a packet and choose Manually Resolve Name where you can assign a name to an IP. Then you won't need to block them out (At least not in the summary pane). 2) Add 2 custom columns to the summary pane: ip.id 3) Always display the column headers as it will be easier to tell which IP is the src or dst. As to your issue it does appear the server is either not receiving the ACK or ignoring it. We would have to know where that capture was taken. answered 29 Jun '15, 09:56  KevinB edited 29 Jun '15, 09:57 Hi Kevin, Yes the screenshot was captured on the server itself, I applied all the suggestion you gave above as below. http://imgur.com/3T8ErmU - Server Side http://imgur.com/IJrFa8v - Client Side Also here are the pcap (I export only the relevant tcp stream) https://goo.gl/5taeqF Hope that's ok. (29 Jun '15, 19:14) Furqan Well the Server is definitely receiving the ACK from the client. Makes no sense why it would be ignoring it. I would focus on where this is occurring. Is it with all clients or only some? Is it repeatable? I noticed that the server has a VMWare NIC. Is that server a VM? (30 Jun '15, 09:43) KevinB This happens to only 1 specific client only while other connecting to this server is not affected and the issue is still occurred intermittently till now. Yes this a VM server running under VMWare. (30 Jun '15, 18:53) Furqan 1 Here is my best guess as to what is happening. The 3way handshake is fine but the server does not receive any TCP data bytes from the client for over 30 seconds so I am guessing that perhaps it retransmits the SYN-ACK to see if the client is still alive. The client's TCP stack immediately DUP-ACKs it but as Christian mentioned the client application process just seems to have gone to sleep. So the server waits a little longer and then FINs the connection. (30 Jun '15, 19:25) KevinB Thanks for your comment, I guess I'll need to talk to the client to check on their side again haha. Thanks again. (30 Jun '15, 20:23) Furqan |
Obviously does the ACK of the 3Way Handshake not reach the server. For that kind you got a spurious retransmission, a retransmitted ACK and in the end the servr closes the connection with a FIN. So somewhere in your network between capture point and server your packet got lost or blocked.
Hi Christian_R,
Thanks for your answer, appreciate it. To clarify things up, the capture point was set on our server itself and this happens intermittent on 1 particular IP only, while other client connected to the same server does not get affected by this at all.
Here is the client side tcpdump.
http://imgur.com/o3cplse
Does this mean the issue is on the client side or my side?. Thanks again.
The issue seems to be that, something at the server side drops the ACK.
So I have looked at the traces: Indeed I think there is no problem with the 3WAY Handshake. To finally check if the server side is allright you can perform a netstat command (e.g. netstat -ano -> on a windows system) and search your active connection. If the connection state is ESTABLISHED then everything went fine on the server side.
I think your client application has fallen into a deep sleep. The normal connection time is 20ms to perform the POST.
But in the failure case the POST Request is transmitted only until a long time past server initiated half close.
Maybe somthing is wrong with the GET-Request of Packet 235 which could be seen in the forest0629.pcap
Thanks for your valuable input, I monitored the connection with netstat and I saw the SYN_RECV then goes into ESTABLISHED and at the same time I monitored the access_log for the http I returned with error 408 the same issue occurred all these time. I guess server side is not the issue then.