This is our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.
0
1

I'm having a problem with a web server, which causes the web client to occasionally throw a Connection refused exception despite the server being up and listening. This seems to happen when the rate of socket establishment from the client is high enough that the client is reusing ephemeral ports roughly every couple minutes.

In packet captures, I note the following unusual sequence around the time of the Connection refused exception:

Client -- SYN -> Server

Client <- ACK -- Server

Client -- RST -> Server

The server replies to SYN with ACK, rather than SYN,ACK. Furthermore, in this ACK, the acknowledgement number is not the SYN sequence number plus one. Rather, it is the same acknowledgement number last sent by the server in the previous socket that used the same client port. This acknowledgement number falls outside the window, thus eliciting a RST response from the client.

Wireshark seq/ack analysis marks the erroneous ACK with [TCP ACKed unseen segment] and shows the RST cause as TCPS_SYN_SENT-Bad_seq

For this service, the server initiates socket close, so all the sockets have the following closing packet sequence:

1) Server -- FIN,ACK -> Client

2) Server <--- ACK ---- Client

3) Server <- FIN,ACK -- Client

4) Server ---- ACK ---> Client

and in the problem cases, the acknowledgement number from (4) above is repeated much later in the server's ACK response to the client's SYN request for a new socket.

The server is running RHEL, and the TIME_WAIT interval is 60 seconds. When I see the problem behavior, the client SYN is sent more than 100 seconds after the closing sequence from the previous time the ephemeral port in the SYN was used, so from the server's point of view the previous connection should be fully closed.

It looks as if the server sometimes loses track that a connection is closed. Any ideas on what might be wrong?

Thanks!

asked 30 Apr '15, 09:37

peconnol's gravatar image

peconnol
6122
accept rate: 0%


Have a look at: http://serverfault.com/questions/303652/time-wait-connections-not-being-cleaned-up-after-timeout-period-expires

In short, there seems to be code in the Linux kernel that limits the amount of TIME_WAIT connections that can be reaped per second. Setting tcp_tw_reuse to 1 will make the kernel accept a new connection on a port that is still in TIME_WAIT.

Also, you can increase the port range that can be used on the client, so it takes longer for the client to reuse the same port.

permanent link

answered 30 Apr '15, 14:15

SYN-bit's gravatar image

SYN-bit ♦♦
17.1k957245
accept rate: 20%

Thanks. I tried this out to see if I can hit the limit on socket reaping but was unable to reproduce the condition described in the referenced URL.

Would setting SO_REUSEADDR within the server have the same effect as setting tcp_tw_reuse? If so I would prefer to set the socket option so as to limit this change to the service and not the whole server.

(04 May '15, 09:29) peconnol
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×139
×55
×6
×6

question asked: 30 Apr '15, 09:37

question was seen: 4,763 times

last updated: 04 May '15, 09:35

p​o​w​e​r​e​d by O​S​Q​A