This is our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

Hi All,

i have a tcpdump. in that there are many rst packets send from host .26 with win = 0. i want to know whether it points to some problem existing in the .26 server application like max process reached or problem in tcp receiver. My .199 server was not able to receive traffic from .26 I want to know was there any issue with .26 server.

This dump was captured in 10.5.207.199 server. We need to analyze traffic b/w 10.5.220.26 and 10.5.207.199. This was a past problem and i am digging to get the RCA for failures. Presently no problems encountered.

Issue is:

  1. requests getting failed from/to 10.5.207.199 server received from 10.5.220.26
  2. very less number of requests received from 10.5.220.26.

I can see RST packets sent from 10.5.220.26 server. There are two types of RST packets sent.

  1. One with valid window length value.
  2. And other with zero window length value.

I feel valid window length value RST packets are not a problem. But RST packets sent with zero window lenght points to problem in 10.5.220.26 application. Zero window length indicates max process reached or problem in tcp receiver. That is the machine is busy or cannot accept any more traffic from other host. This may lead to delays or connection problems.

I need confirmation that this thinking is right and there was problem in 10.5.220.26 server. Please confirm.

I can see source machine ip as 10.5.220.26 in the tcpdump sending RST messages.

The tcpdump file is available at Cloudhshark.

asked 22 Oct '17, 00:51

HARPREET's gravatar image

HARPREET
6223
accept rate: 0%

edited 22 Oct '17, 03:33

sindy's gravatar image

sindy
6.0k4851


My conclusion would be that the RST packets with zero window length are not a symptom of overload of the .26 machine.

Use the following display filter:

tcp.flags.reset == 1 and tcp.window_size == 0 and ip.addr == 10.5.220.26 and tcp.seq > 2

This will show you only RST packets which aren't the only ones belonging to their TCP session in the capture and where the client->server data of the session are present in the capture, so you have a chance to see what had happened in their respective sessions before the RST packets came.

Now find sessions where the tcp.seq in the RST packet is below 3000. These seem to be conversations where both the SOAP request and response have been captured and where only a single session over a given socket pair has been captured; where the tcp.seq in the RST packet is much higher, you'll see that the socket pair has been reused.

Now find the tcp.stream value for any of them (e.g., 85) and use this value to show the whole stream, using a display filter tcp.stream == 85.

You'll see an almost normal exchange - the client sets up the session and sends a SOAP request, and it sends a FIN after it receives an ACK for the last segment of the request, indicating that it is not going to send anything else using this session. What is strange is that the client (.26) sends the packet with FIN exactly 10 seconds after getting an ACK to the last segment of the request, and that the server (.199) doesn't start sending the response until it receives that FIN from the client.

After the server (.199) finishes sending its response, it sends its own FIN. Normally, the client should send an ACK to that FIN and the session would be completely closed. Instead, it sends a RST (with window size = 0, but that's not important because the server has already sent FIN so it is not going to send data any more). That may be a way to avoid the tcp wait state which normally follows peaceful closure of a TCP session to provide a guard period to avoid confusion of packets belonging to two distinct TCP sessions using the same socket pair.

If you filter using tcp.stream == 73, you'll see a slightly different picture (several SOAP requests and responses in a single TCP session) but still with the 10 seconds gap before the client sends its FIN and still with the client responding with RST to server's FIN.

Why I don't think that the RST with window size 0 indicates an overload of .26:

  • because the .26 sends window size 0 after receiving FIN from the opposite end so it is actually irrelevant,
  • because the .26 is a client in the sessions so if it would be overloaded, it would stop opening new sessions at first place.

On the other hand, I do admit that the capture contains also cases where, in an otherwise same pattern, the window size in client's RST is different from 0.

The 10 second gap before sending the FIN seems to be a guard period for eventual other requests to come from the application. What is strange is that although the last segment of the request bears a PSH flag, the server does not respond immediately but only after receiving the FIN flag. This is only the case for the last request in the session, so maybe it is related to the particular type of that request?

permanent link

answered 22 Oct '17, 05:21

sindy's gravatar image

sindy
6.0k4851
accept rate: 24%

so it seems everything is normal, there r tcp and http sessions formed. data is exchanged b/w 26 and 199 server. I have also raised this towards my application side that is some problem in 199 server application. as I saw some Oracle errors in this tcpdump. but I also got "possible syn flooding at Port 44006. check snmp counters." messages in my 199 server so was concerned that some client is creating problems. wht do you think is there any client which sends delayed ayk messages after receiving syn ayk messages from server 199?

(22 Oct '17, 06:49) HARPREET

Answering that additional question in detail might require some programming (using MATE or Lua) to add meta-fields to the sessions' packets, allowing to use display filtering to highlight unusual sessions. However, a display filter tcp.analysis.reused_ports is enough to highlight some cases.

Use display filter tcp.port == 43239 to show one of them. The client sends a SYN packet (note that in this capture the very first one is lost), but the server responds with an ACK with totally unrelated tcp.ack number (look at absolute seq/ack numbering), most likely because it still considers the received SYN packet to be part of the previous session established from the same ephemeral port at client side. Consequently the client sends a RST because the received tcp.ack value is inacceptable for it.

I belive the root cause is the incompatibility between the client's idea that sending a RST in response to server's FIN will clear the TIME_WAIT state at server side and server's moving the session to TIME_WAIT state anyway. You can see that sooner or later the server does accept the next SYN from the same client port and answers it with a matching tcp.ack, making the client continue the session normally.

I'm not an expert on tcp session closure so I don't know which one of the two approaches to RST after second FIN is right. However, it seems that the rate of tcp sessions opening from .26 to .199 per unit of time is higher than the number of available ephemeral ports divided by the 120 seconds of (typical) TIME_WAIT duration.

  • Reducing TIME_WAIT duration at server side may seem the most straightforward solution but it may be dangerous if some clients connect via lossy paths with large transmission delays.
  • Binding additional instances of the server application to several other ports than 44006 and letting the client distribute the load among them might be a better approach.
  • As for me, the best approach would be not to close the already established TCP sessions and reuse them for multiple request-response exchanges, but I don't know whether SOAP doesn't use the TCP session as a context linking together related requests.
(22 Oct '17, 07:35) sindy
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×81
×55
×29
×16
×9

question asked: 22 Oct '17, 00:51

question was seen: 3,316 times

last updated: 22 Oct '17, 07:35

p​o​w​e​r​e​d by O​S​Q​A