Hi All, i have a tcpdump. in that there are many rst packets send from host .26 with win = 0. i want to know whether it points to some problem existing in the .26 server application like max process reached or problem in tcp receiver. My .199 server was not able to receive traffic from .26 I want to know was there any issue with .26 server. This dump was captured in 10.5.207.199 server. We need to analyze traffic b/w 10.5.220.26 and 10.5.207.199. This was a past problem and i am digging to get the RCA for failures. Presently no problems encountered. Issue is:
I can see RST packets sent from 10.5.220.26 server. There are two types of RST packets sent.
I feel valid window length value RST packets are not a problem. But RST packets sent with zero window lenght points to problem in 10.5.220.26 application. Zero window length indicates max process reached or problem in tcp receiver. That is the machine is busy or cannot accept any more traffic from other host. This may lead to delays or connection problems. I need confirmation that this thinking is right and there was problem in 10.5.220.26 server. Please confirm. I can see source machine ip as 10.5.220.26 in the tcpdump sending RST messages. The tcpdump file is available at Cloudhshark. asked 22 Oct '17, 00:51 HARPREET edited 22 Oct '17, 03:33 sindy |
One Answer:
My conclusion would be that the RST packets with zero window length are not a symptom of overload of the .26 machine. Use the following display filter:
This will show you only RST packets which aren't the only ones belonging to their TCP session in the capture and where the client->server data of the session are present in the capture, so you have a chance to see what had happened in their respective sessions before the RST packets came. Now find sessions where the Now find the You'll see an almost normal exchange - the client sets up the session and sends a SOAP request, and it sends a FIN after it receives an ACK for the last segment of the request, indicating that it is not going to send anything else using this session. What is strange is that the client (.26) sends the packet with FIN exactly 10 seconds after getting an ACK to the last segment of the request, and that the server (.199) doesn't start sending the response until it receives that FIN from the client. After the server (.199) finishes sending its response, it sends its own FIN. Normally, the client should send an ACK to that FIN and the session would be completely closed. Instead, it sends a RST (with window size = 0, but that's not important because the server has already sent FIN so it is not going to send data any more). That may be a way to avoid the tcp wait state which normally follows peaceful closure of a TCP session to provide a guard period to avoid confusion of packets belonging to two distinct TCP sessions using the same socket pair. If you filter using Why I don't think that the RST with window size 0 indicates an overload of .26:
On the other hand, I do admit that the capture contains also cases where, in an otherwise same pattern, the window size in client's RST is different from 0. The 10 second gap before sending the FIN seems to be a guard period for eventual other requests to come from the application. What is strange is that although the last segment of the request bears a PSH flag, the server does not respond immediately but only after receiving the FIN flag. This is only the case for the last request in the session, so maybe it is related to the particular type of that request? answered 22 Oct '17, 05:21 sindy |
so it seems everything is normal, there r tcp and http sessions formed. data is exchanged b/w 26 and 199 server. I have also raised this towards my application side that is some problem in 199 server application. as I saw some Oracle errors in this tcpdump. but I also got "possible syn flooding at Port 44006. check snmp counters." messages in my 199 server so was concerned that some client is creating problems. wht do you think is there any client which sends delayed ayk messages after receiving syn ayk messages from server 199?
Answering that additional question in detail might require some programming (using MATE or Lua) to add meta-fields to the sessions' packets, allowing to use display filtering to highlight unusual sessions. However, a display filter
tcp.analysis.reused_ports
is enough to highlight some cases.Use display filter
tcp.port == 43239
to show one of them. The client sends a SYN packet (note that in this capture the very first one is lost), but the server responds with an ACK with totally unrelatedtcp.ack
number (look at absolute seq/ack numbering), most likely because it still considers the received SYN packet to be part of the previous session established from the same ephemeral port at client side. Consequently the client sends a RST because the receivedtcp.ack
value is inacceptable for it.I belive the root cause is the incompatibility between the client's idea that sending a RST in response to server's FIN will clear the TIME_WAIT state at server side and server's moving the session to TIME_WAIT state anyway. You can see that sooner or later the server does accept the next SYN from the same client port and answers it with a matching
tcp.ack
, making the client continue the session normally.I'm not an expert on tcp session closure so I don't know which one of the two approaches to RST after second FIN is right. However, it seems that the rate of tcp sessions opening from .26 to .199 per unit of time is higher than the number of available ephemeral ports divided by the 120 seconds of (typical) TIME_WAIT duration.