tcp reset by the server due to 2 consecutive SYNs

Question

Hi,

I have a sequence of tcp connection establishment as follows:

client sends a syn to server
client do not hear a response within 2.996seconds
client initiates a second syn to server
server acknowledges for the syn. not sure if its for the first syn or second syn. but looking at the timestamp its matching with the second syn
client receives a RST,ACK.

I am not sure why i receive a TCP reset. Is there a problem in client or server ? is the server recognizing it as a syn flood ? destination is a VIP (might be a loadbalancer cum firewall). I am actually an application's guy with very limited knowledge on tcp/networking stuff. Could you pls help me out here as i have no clue of whether its an issue with client or server.

SYN goes with flags SYN, ECN, CWR for both SYNs.

Thanks, resolver

Answer 1

0

SYN goes with flags SYN, ECN, CWR for both SYNs.

from RFC 3360:

   
   Unfortunately, a number of firewalls and load-balancers in the
   current Internet send a reset in response to a TCP SYN packet that
   use flags from the Reserved field in the TCP header.  Section 3 below
   discusses the specific example of firewalls that send resets in
   response to TCP SYN packets from ECN-capable hosts.

well, that was in 2002. Anyway, as you mentioned a loadbalancer and/or a firewall, I would try to check that with the device admins in charge.

Can you disable the use of ECN on your client (just for a test)?

http://en.wikipedia.org/wiki/Explicit_Congestion_Notification#ECN_support_in_TCP_by_hosts

Regards
Kurt

answered 27 Dec '12, 23:09

Kurt Knochner ♦
24.8k●10●39●237
accept rate: 15%

edited 27 Dec '12, 23:19

Hi Kurt,

Thanks a lot for your response. I will disable the ECN and test it.

Is there a way that i can post the pcap file to you. if you dont mind pls share your email id. Following is the sequence for one of the resets.

No.       Time         Source      Destination           Protocol Length Info
11931 6218.308766      SrcHst        DstHost           TCP      74     13358 > https [SYN, ECN, CWR] Seq=0 Win=5840 Len=0 MSS=1460 SACK_PERM=1 TSval=145993636 TSecr=0 WS=256
11972 6221.305267      SrcHst        DstHost           TCP      74     13358 > https [SYN, ECN, CWR] Seq=0 Win=5840 Len=0 MSS=1460 SACK_PERM=1 TSval=145994386 TSecr=0 WS=256
11973 6221.328336      DstHost       SrcHst            TCP      60     https > 13358 [ACK] Seq=1 Ack=1 Win=12288 Len=0
11974 6225.741699       DstHost       SrcHst            TCP      60     https > 13358 [RST, ACK] Seq=1 Ack=1 Win=12288 Len=0

I will go through the links you have shared. What are the consequences of disabling the ECN at the device level ?

Not able to post the above details via “add new comment” button hence posting in the Your answer text area. if you would like to move to my comment in any way pls do so.

Thanks a lot again.

(28 Dec ‘12, 02:27) synresolver

Thanks a lot for your response. I will disable the ECN and test it.

O.K. what is your client OS?

Is there a way that i can post the pcap file to you. if you dont mind pls share your email id. Following is the sequence for one of the resets.

If you can upload it somewhere (google docs, one-click filehoster) I would prefer that. Otherwise you can find my e-mail address in my profile (just click on my name).

I will go through the links you have shared. What are the consequences of disabling the ECN at the device level ?

If ECN causes the problem, then the ‘consequence’ would be that it works ;-)

The ‘bad’ consequences are minimal (or none), if your network connection is otherwise stable (not too much packet loss).

(28 Dec ‘12, 03:01) Kurt Knochner ♦

You can post capture files to www.cloudshark.org, but beware that you don’t publicly share capture files with sensitive data in it (you can change IP addresses with bittwist or tcprewrite).

(28 Dec ‘12, 03:01) SYN-bit ♦♦

Hi Kurt,

Sorry for delayed response due to the holdiays. I have sent you the pcap file. If you filter with ip.addr == 20.20.20.20 && tcp.port == 53810 you can see the SYN, ECN, CWR are being set but at the IP level the differentiated services flag states that its not ECN-Capable support. So, not sure if this is completely because of ECN flag.

OS - is IBM’s DataPower XI 50 server.

For the other successful tcp connections, i can see these so network team is ruling out this is the root cause, hence we are not able to disable the ECN and test.

synbit- thanks for suggesting the tools and i modified the capture files to certain extent so that the data is being scrambled.

thanks again.

(02 Jan ‘13, 05:24) synresolver

O.K. I checked the pcap file you sent me via email. Please apply the following filter:

tcp.flags.syn eq 1 and (tcp.stream eq 16 or tcp.stream eq 632)

As you can see, there are HTTPS connections that do work with ECN (and CWR) set. Stream #16 works, whereas stream #632 (similar connection parameters) does not work.

Then please apply this filter:

ip.addr eq 20.20.20.20 and ((tcp.flags.syn eq 1 and tcp.flags.ecn eq 1) or tcp.flags.reset eq 1)

As you can see, most (similar) connections do work, just a few show that 'silly' behavior.

So, I don't think that ECN (and CWR) is the problem. Now I believe it might be a bug/problem with the device between 10.10.10.10 and 20.20.20.20 (if there is one) or the server 20.20.20.20 itself. However, with just a capture from 10.10.10.10, it is hard to make any assumption.

Recommendation: Please try to capture at two points in parallel. One at 10.10.10.10 and one one 20.20.20.20 (if that is the sever), or after 20.20.20.20 if that is the firewall/loadbalancer you mentioned.

BTW: do you know what type of firewall/loadbalancer is in use?

(02 Jan '13, 06:26) Kurt Knochner ♦

server acknowledges for the syn. not sure if its for the first syn or second syn. but looking at the timestamp its matching with the second syn

That's not quite correct, which also make me believe it's a bug as mentioned above.

I think I can post the following output, as its totally anonymous.

   No.     Time        Source                Destination           Protocol Length Info
   7113 3677.285242 10.10.10.10           20.20.20.20           TCP      74     53810 > https [SYN, ECN, CWR] Seq=0 Win=5840 Len=0 MSS=1460 SACK_PERM=1 TSval=145592423 TSecr=0 WS=256
7151 3680.281717 10.10.10.10           20.20.20.20           TCP      74     53810 > https [SYN, ECN, CWR] Seq=0 Win=5840 Len=0 MSS=1460 SACK_PERM=1 TSval=145593173 TSecr=0 WS=256
7152 3680.312861 20.20.20.20           10.10.10.10           TCP      60     https > 53810 [ACK] Seq=1 Ack=1 Win=12288 Len=0
7153 3684.553901 20.20.20.20           10.10.10.10           TCP      60     https > 53810 [RST, ACK] Seq=1 Ack=1 Win=12288 Len=0

As you can see:

Frame #7113: 10.10.10.10 -> 20.20.20.20 SYN,ECN,CWR
Frame #7151: 10.10.10.10 -> 20.20.20.20 SYN,ECN,CWR

then

Frame #7152: 20.20.20.20 -> 10.10.10.10 only ACK, no SYN/ACK. So this is not a valid reply.
Frame #7153: 20.20.20.20 -> 10.10.10.10 RST

very strange and possibly a bug in the firmware of the firewall/loadbalancer (or server). See my previous comment.

(02 Jan '13, 13:06) Kurt Knochner ♦

showing 5 of 6 show 1 more comments

Answer 2

is there a way to adjust your tcp settings so that it doesn't send the second SYN until more time has passed? If it's consistently taking ~2 seconds to respond to SYN, wait 3 before you retransmit, etc... likewise, you might be able to use HPing or SCAPY to send a single SYN packet on it's own and just see what happens. If you still get no reply or an RST, then it's an issue at/past whatever loadbalancer/firewall is in place. I've only rarely used those programs, so I can't remember the relevent commands, but there's plenty of info online.