Huge Number of TCP Reset Error

You are right that the SYN should come only once in the beginning of session, yet I am not a TCP expert either. But:

• any packet in any direction may be lost and thus retransmitted, and the travel time of any two packets between the same source and destination may differ significantly. So you cannot exclude that the initial SYN would come twice or more times if your first SYN+ACK has not reached the session initiator fast enough. And due to the variable delay, you may have already ACKed a received SYN, the remote party may have received your ACK and sent to you the first data packet (or even more as ACK for every single packet is not mandatory), and still a SYN retransmission may arrive.
• the ACK is not just a boolean value (nothing like NAK exists in tcp); instead, the sender of the ACK indicates to the remote party the order number of the byte it has successfully received.

So open your anonymized capture and use display filter (ip.src == 10.10.10.10 and tcp.srcport == 44157 and tcp.dstport == 80) or (tcp.srcport == 80 and tcp.dstport == 44157 and ip.dst == 10.10.10.10). You shall see that

• the first tcp session has started with a SYN in frame 61303 and ended with ACK to FIN in frames 61315, 61316.
• it then took about 32 seconds until the client side has used the same socket for a new session starting with SYN in frame 101255 and ending with ACK to FIN in frames 101333, 101334.
• and it took another about 36 seconds until another SYN has arrived from the same socket - frame 148368, which has been evaluated as part of the previous session and as such ACKed with a local sequence number from that session, which the client has evaluated as an error and has responded with an RST.

I was looking for some difference between the two cases and it is not very convincing:

• the absolute sequence number in frame 101255 was 0x90ab3f05 which is slightly lower than sequence number 0x91d3af78 of the FIN frame 61315,
• the absolute sequence number in frame 148368 was 0x4b081d10 which is much lower than sequence number 0x90ab44f3 of the FIN frame 101334.

So we can see that the tcp stack in your server treats the two quite similar cases differently but I don't dare to guess why exactly.

... to be continued

I wanted the capture from client side because I've expected to see some lost FIN packets, and I also didn't know you knew the public IP address of your client. As it looks now, the capture from their side won't bring too much additional information.

My opinion is that you have to try to use the settings suggested at the link I've sent you before and hope that they help, because otherwise you would have to (let someone?) dive deep into the tcp stack of your server's kernel and change its treatment of remote socket reuse.

Hm, I am afraid you will need a kernel recompilation... see this thread for explanation why.

In short, the window during which the kernel tcp stack still treats new packets from the same socket as late arrivals is 60 seconds and it is not configurable: net/tcp/tcp.h says

#define TCP_TIMEWAIT_LEN 6000 /* How long to wait to sucessfully close the socket, about 60 seconds. */

And please do not ask me why your server does not have the same issue with the first reuse in the capture.

Your other option would be not to touch the server and insert a load balancer in front of it, which would spread the traffic among several servers or, if you feel brave, just among several ports of your current server to which you would attach the httpd.

The assumption is that a load balancer's tcp stack should be better prepared for this (or not stateful, just passing the packets through while changing the tcp port number) so at each of the server's sockets, the SYN of the next connection would not arrive earlier than 60 seconds after the FIN/ACK/FIN/ACK of the previous one.

And maybe the simplest to implement if you encounter the issue only with that single client: could you ask them to use more public addresses at their NAT device? Looking at the reuse rate in this capture, three public IPs should be enough to get rid of the problem. If they would also broaden the range of ports the NAT device is allowed to use (the capture shows that they use a range from 39xxx through 60xxx, which quite matches your 20000 connections in TIME WAIT state), just two public IPs would be enough for the traffic volume seen in the capture.

Yet not knowing the client application, I cannot predict whether if the connections would stop failing, their total count would decrease because less retries would be necessary, or whether it would increase because the client would be able to pass more requests through the (then wider) bottleneck of their NAT device.

thanks for the analysis sindy, i'm going through it and trying to understand our feasible solution. Few questions

1. we are using loadbalancer HAproxy. just wanna understand in this case loadbalancer will temporarily keep all tcp segements in own buffer, and after confirming all timers, sequence, out of order frames. it will forward to server? is it how it will work?

2. another solution might be asking the client to extend the port numbers range (1025 - 65534). to overcome reuse problem.

3. TIMEWAIT is interesting.. wanna know is it TCP_TW_REUSE? both are same? altering this key might solve socket exhaustion problem but i think it will increase the resets.

what your say?

"and it took another about 36 seconds until another SYN has arrived from the same socket - frame 148368, which has been evaluated as part of the previous session and as such ACKed with a local sequence number from that session, which the client has evaluated as an error and has responded with an RST."

within 75 seconds i see this socket used three times.. why? and i have a question. isn't SYN = LAST RECEIVED ACK or 0? so lets say we generated 3 SYN for a session and our server responded to second one, the 1st one lingering in the path. now i wanna know the SEQ NUM for all SYN would be same or different? if it is same then isn't any mechanism in TCP STACK where it can reject the already ACKed SYN by checking SEQ number. and if all 3 SYN are different then wouldn't it be considered as 3 different connections? hmm... i'm thinking SACK. but its already ON.

I say few responses:

1. We are at Wireshark Q&A, aren't we. So take a capture on the machine running the HAProxy (simultaneously on all interfaces that may be involved) and you'll see the answer. Or ask the HAProxy authors. I could only suppose. But your mentioning of HAProxy changes a lot of things. Where exactly the capture you've posted has been taken? Between the client and HAProxy or between HAProxy and the server? Maybe the 20000 ports pool is set on your HAProxy and not on the client side? Can you provide a drawing of the complete architecture of your site (covering the whole path between the interface providing connection to the internet and the server)?

2. yes, that is why I've recommended already, but it may not be enough, as I've also already written: Currently, the machine from which the requests are coming is cyclically reusing some 20000 ports. If they start using a pool of 63000 ports, it may solve the problem of mistreatment of new sessions, but only if it is the only problem in the whole chain. Because the capture on your end only shows the requests that made it from the real client to your server through the limitation of the 20000 available ports somewhere between the real client and your server. There may have been a ton of connection attempts that have been rejected already at client side due to unavailability of free port, and these would reach your site after the number of ports would get tripled.

within 75 seconds i see this socket used three times.. why?

because during that 75 seconds, the available pool of ~20000 sockets has been cycled through three times, i.e. the client(s) has attempted to open (at least!) ~60000 tcp sessions during that time. At least because there may have been even more attempts but we cannot see them as they haven't made it through the 20000 ports limit.

now i wanna know the SEQ NUM for all SYN would be same or different?

See the TCP RFC, but I haven't found anything regarding treatment of mid-session SYN there. In any case, the very meaning of a SYN indicator set to 1 in a packet is to declare that the absolute SEQ NUM in that packet is the initial (reference) sequence number for the session, i.e. a relative 0. So it would be a very bad idea to use different absolute SEQ NUM in the retransmissions of this packet.

Just out of curiosity have you set tw_recycle =1? Maybe it has been written here, then I missed it.

Personal I do think, the smartest solution is to add one or more client IP's as @sindy suggested, because it is independent of operating system implementations.

But here is another nice article: http://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html#netipv4tcp_tw_recycle

@arjun-singh, forget what I wrote about tcp_tw_reuse. It is only relevant to outgoing session which is not your case. The article which @Christian_R has mentioned gives a much clearer view than the one I've given before. I even doubt whether I've understood properly the meaning of tcp_fin_timeout.

@Christian_R, in the great article you've mentioned (I think one would have a hard time trying to find something to add to it, thank you for the link!) there is one important paragraph related to use of tcp_tw_recycle:

When the remote host is in fact a NAT device, the condition on timestamps will forbid all of the hosts except one behind the NAT device to connect during one minute because they do not share the same timestamp clock. In doubt, this is far better to disable this option since it leads to difficult to detect and difficult to diagnose problems.

So until @arjun_singh tells us whether there is a NAT on client side, we cannot say whether use of tcp_tw_recycle would help or actually make it worse.

client agreed upon sharing captures...so will revert soon... in the meantime i'm sharing topology at our end.

http://s000.tinyupload.com/?file_id=98475239441193171483

hi guys, first of all thank you for being with me..

last night we face sudden surge of spurious re-transmission on our both servers. I got hands on live captures for both side. After analysing it i found that sindy was right client has NAT enabled.. 'cos i'm seeing private ip in capture. but i didn't allowed tcp_tw_recycle and its value is 0 on our loadbalancer.. I'm attaching both captures here.. please have a look..
our ip_local_port_range is 32768-61000,
tcp_tw_recycle = 0
tcp_tw_reuse =
tcp_fin_timeout = 60

Our capture >> http://www93.zippyshare.com/v/zd8f670O/file.html

Client Capture >> http://www93.zippyshare.com/v/busirUxE/file.html

10.10.10.10 >>> client public IP
10.248.187.181 >> Our Internal IP (private)
192.168.201.56 >> client Internal IP (private)
50.60.70.80 >> Our Public IP

@Sindy we tried with VPN first place, but it didn't work out. there were much more timeouts than public. but my first priority is VPN and i'll go for it after this issue get solved. I'm putting IP address / Port range extension suggestion on table tonight. so Lets see.. :) till then can you guys tell me why these spurious tranmission happend?? it came suddenly for 5 mins then again everything went normal.

My another question which answer i couldn't find anywhere in your links clearly is, in tcp_tw_reuse server can reuse the ports in TIME_WAIT state, now if a current session is not over and a socket is reused with new request how tcp gonna know that its NEW not OLD one when its session is still running (in TIME_WAIT counter). and if i say timestamps, then how can we so sure about synchronization of time stamps of thrice servers, as you already quoted a 3 timestamp mismatch.. where our LB got confused?

thanks

Hi Arjun, I cannot download huge captures on the road, but to answer your questions which don't require analysis of the new captures:

in tcp_tw_reuse server can reuse the ports in TIME_WAIT state

let's remind that a "server" can have two meanings:

• a role in a session (a "client" is the party which actively opens the session, i.e. sends the initial SYN in case of tcp session, while a "server" is the party which receives the initial SYN and responds with SYN, ACK)

• a powerful hardware intended to serve service requests from many clients.

Having said that: I understand the article found by @Christian_R that tcp_tw_reuse is only meaningful for a tcp session client, as it permits it to open a new session towards the same remote socket from the same local socket while the previous session using that socket pair is still in TIME_WAIT state. In another words, enabling tcp_tw_reuse has no effect on sessions coming to your servers from tcp clients. Going deeper into it (an not related to your current investigation), tcp_tw_reuse on client side without knowing that tcp_tw_recycle is enabled on the server side is a bad idea because an attempt of the client with tcp_tw_reuse enabled to establish a session reusing the same socket pair will fail if tcp_tw_recycle is not enabled at the server side. Make your own conclusion about how useful these settings actually are, given that they apply for the whole tcp stack and cannot be narrowed to apply only to a list of remote IP addresses.

sindy?? how moving to iPv6 gonna help us. I changed the tcp_fin_timeout at our LB to 10. i'm analyzing the traffic, but couldn't see much effect.

@Arjun Singh

Your "answer" has been converted to a comment as that's how this site works. Please read the FAQ for more information.