Hi, I hope someone here can help. I have seen a strange problem where our server seems to be retransmitting the segment prior to the one being requested by the Duplicate ACKs sent by the client. The Duplicate ACK is requesting re-transmission: Acknowledgement number: 422281 (relative ack number) Sometimes I see an ICMP packet being sent by the server after receipt of the Duplicate ACK, usually in response to the first five of these Duplicate ACKs, but can be later as well, in this particular case the last five received too : Then the server retransmit sends the previous sequence: This continues until the conversation peters out after about a minute. Has anyone seen anything like this before? asked 25 Jun '12, 04:46  Christian |
One Answer:
Where did you capture? I guess at the client side. Do you have verified the behavior by capturing on the server side as well? I suspect there's a firewall or similar device with ACLs in the communication path between client and server (the ICMP administratively prohibited indicates a "reject" rule existing somewhere). If there is a firewall you need to verify first that the server actually gets the same TCP details that the client sent. A firewall might modify the TCP header in flight so that the server does not actually get the ACK numbers the client sent. Maybe it even removes the SACK options for whatever reason. So you need to make sure that the server sends data from before the last ACK number by verifying that behavior as close to the server as possible (but please not by capturing on the server itself, because that could give funny results, too). Don't forget to inspect and compare the SYN-SYN/ACK sequence at the beginning of the communication on both sides, because you might see that some TCP option negotiation is being influenced as well. answered 25 Jun '12, 06:09  Jasper ♦♦ Thanks Jasper, I managed to get traces simultaneously from both client and server. The problem is the web server is virtual, in a hosted location in London and the client is accessing through the internet so I am restricted to captures on the endpoints. I'm trying to pinpoint what I think is problem in a local authorities ISP making it even harder to get details about what is happening to the packets en-route. There are some differences in the SYN, SYN-ACK packets, the MSS is different at either end, the server MSS is 1380, the clients is 1460 , TTL is different too, 121 to 128. Otherwise the rest seems the same The ICMP is being sent by the server, and seems to relate to some of the SACK packets with the rubbish SLE/SRE (ports 80 and 443 are open). I was wondering whether the ICMP was produced because the SLE/SRE can't possibly be correct for the file being transmitted? I can see no other purpose for this ICMP as the host is reachable. This ICMP is also not reaching the client. I know the ISP blocks ICMP through its network on 'security' grounds. (25 Jun '12, 07:33) Christian what is the payload of the ICMP packet? What is your OS at the server? Is there any security software installed on that server? Reason: That ICMP message is rather seldom (even for firewalls), so there must be a reason for this, IF it belongs to your problem (TCP connection), hence the question about the payload. (26 Jun '12, 00:14) Kurt Knochner ♦ regarding the broken SRE/SLE values. Does this happen
(26 Jun '12, 01:38) Kurt Knochner ♦ CentOS 6.2 patched to last month, no extra security, just the default firewall allowing ssh, http and https set using system-config-firewall-tui. The ping contains the nonsense SLE/SRE from the previously received SACK packet in Options. SACK: 1186682302-1186685062 340552 (server) Arrival Time: Jun 20, 2012 15:56:38.036203000 BST Epoch Time: 1340204198.036203000 seconds 96194 (client) Arrival Time: Jun 20, 2012 15:56:38.021608000 BST Epoch Time: 1340204198.021608000 seconds (26 Jun '12, 01:54) Christian can you please upload just those two packets to cloudshark.org? I want to take a look at the SACK options in detail, not just the "Info Summary". You can change the IP addresses with 'tcprewrite' HINT: You cannot delete anonymous uploads to cloudshark! (26 Jun '12, 01:59) Kurt Knochner ♦ With every connection after data is lost in transit, SLE/SRE is always broken for connections passing through this service provider. The packets passing through the network are NATed by something and I suspect this device. As for the ICMP, I think it is trying to communicate the root of the problem, but as the network in question drops all ICMP "for security reasons", the packet mangling box is never getting to hear about it. (26 Jun '12, 02:13) Christian Are the packets NATed at the client side (Firewall/Router) or somewhere within the ISP network? If at the client side, is the NATed source port the same as in the packet at the server side (possible double NAT)? (26 Jun '12, 02:20) Kurt Knochner ♦ Had to do this with two files, the first for the server I also included the ICMP response: http://cloudshark.org/captures/39ca99695903 The second for the client: http://cloudshark.org/captures/578b446be516 (26 Jun '12, 02:32) Christian I don't know the details of the NATing on their network, but it is not beyond the realms of possibility that it is a double NAT, however I notice their clients have 10.0.0.0 series IPs rather than the more usual 192.168 or 176. so potentially they could be just one big network. (26 Jun '12, 02:36) Christian tcprewrite seems to have messed up the SACK numbers in the ICMP packet. In my original file the SACK data contained in Options is as follows: (26 Jun '12, 03:11) Christian
the values in the ICMP packet are the real (absolute SEQ numbers). Wireshark uses relative SEQ numbers per default. If you change that, the values in the TCP packet and the ICMP packet are identical.
(26 Jun '12, 03:40) Kurt Knochner ♦
there could be WAN accelerators in place that modify the values due to a bug. Without knowledge of the network architecture, it can be virtually anything :-( It would be interesting to see if you can find a "pattern" for the SRE/SLE value changes (client versus server), like: always same delta, slightly changing delta, etc. You can do that with tshark: -T fields -e tcp.options.sack_le -e tcp.options.sack_re (26 Jun '12, 03:43) Kurt Knochner ♦ Thanks for your expert help, I'm going to recommend to the chaps in the ISP that they review their blanket ban on ICMP and investigate what is corrupting the SLE/SRE data in the packets. I don't know how they manage to get anything across their network! (26 Jun '12, 03:49) Christian If you provide the information about SRE/SLE to the ISP, give them the absolute SEQ numbers, not the relative ones. Good luck! (26 Jun '12, 03:51) Kurt Knochner ♦ showing 5 of 14 show 9 more comments |
I should add that the ACK and duplicates mentioned above are SACK packets with the SRE incrementing with each DupACK until all in-flight data is received and the server starts to retransmit the packet in the sequence prior to number requested.
Further I suspect that something between client and server is meddling with the packets as the SLE and SRE transmitted by the client in the SACK is not the same as the SLE and SRE seen by the server.
Sent by client:
96194 TCP 66 tksocket > http [ACK] Seq=883 Ack=422281 Win=65535 Len=0 SLE=447121 SRE=448501
Received by server:
340552 TCP 66 25882 > http [ACK] Seq=883 Ack=422281 Win=65535 Len=0 SLE=1186682302 SRE=1186683682