Can ACK in segment be greater than LE and RE? Actually I observe such behaviour a lot [TCP Dup ACK 6126#1] 55160 > 80 [ACK] Seq=3817 Ack=4157995 Win=123236 Len=0 SLE=4149831 SRE=4151291 We're trying to troubleshoot strange issue of client-browsers sometimes receiving crashed data over HTTP. Client and server both have 10g NICs in pure 10g environment. Client - VM (esxi). Whats the proper way of troubleshooting such things? I have uploaded one of such traces http://cloudshark.org/captures/7a5801e7fa42 This trace captured 2 successfull http-GETs and then 1 failed - broken picture in browsers. And strange sacks are in frames 3688,3699, for example. asked 28 Jan '13, 21:23  v_paranoid edited 30 Jan '13, 21:01 |
2 Answers:
Yes, an ACK in a TCP packet can be greater than the SACK LE/RE values. This is lesser known extension to the SACK mechanism specified in RFC 2883, which introduces the so called D-SACKs (for "Duplicate SACK"). The intention is to have a mechanism with which the receiver can tell the sender that segments were received again, while the original segment had already been received AND acknowledged. Sort of "why are you sending me THIS again?"-notification, that might help the sender to adjust his sending algorithm. Question remains - is this a case of D-SACKs here, or is something bad going on? Let's see. I think your problem has to do with packet 3688: it acknowledges (relative) number 811165, while the acknowledge before that (in packet 3687) acknowledged 803001 - that is a jump of 8164 bytes. That would mean that we should have seen packets carrying this much data, but that is not the case: In packet 3686 we got sequence number 803001, and that packet gets acked in 3687. And the interesting thing is that the next packet (in #3689) we see from the server has sequence number 803001, and going up like clockwork in the next couple of packets. Everything would be perfectly fine if we remove packet 3688 from the list, because without it, everything works as it should. In packet 3699 we see the D-SACKs coming in - which makes sense now: the receiver has (by mistake, as it seems) increased it's ACK number to 811165, and the sender is still busy sending the real segments that have sequence numbers lesser than that value. So the receiver notifies the sender "I got this already" - which in this case is a lie from what we can tell from the trace file. It didn't get the "original" packets, it only made a jump in it's ack number tracking. Final verdict: I can't tell you what the receiver thinks he's doing, but the ack number jump in frame 3688 looks bogus to me (I verified with absolute sequence numbers, it's crucial to do that, but it gives the same result in this case). I also do not know why the jpg gets damaged, but if the TCP stack is doing funny things like what we see in frame 3688 I can guess that it probably fouled up the data assembly and transmission to the application layer on the receiving side. Because in the jump of 8164 bytes that apparently weren't received but marked as such you'd have a huge gap in your image data - and if the receiver discards the incoming "real" segments (which it probably did, but that's guesswork) you'll have a "hole" in your data. Boom, JPG dead. answered 31 Jan '13, 14:31  Jasper ♦♦ edited 31 Jan '13, 15:31 |
No, the ACK should not be greater than SLR or SRE. It should be: ACK < SLE < SRE If there are multiple SACK blocks in the packet, then it should be: ACK < SLE1 < SRE1 < SLE2 < SRE2 ... etc. The numbers should increase from left to right, and there should be a gap between the ACK and SLE. In this case, ACK and SLE overlap; there is no gap. It looks like the TCP/IP stack on 212.192.64.74 (or on some intermediate security device) gets confused and uses incorrect numbers. Frame 3687, with ACK 803001 (using relative sequence numbers), is an ACK to frame 3686. It is immediately followed by ACK 811165 in frame 3688. Frame 3688 appears to be just a window update, but with an incorrect ACK number. TCP/IP normally ACKs full segments. The ACK for one packet is the sequence number expected on the next packet. 811165 does not correspond to the start of any packet in this stream. 811165 falls in the middle of frame 3694; 864 bytes in, to be exact. Notice also that the SACK block keeps moving. In frame 3699, SLE is 803001 and SRE is 804461. In frame 3700, SLE is 804461 and SRE 805921. In frame 3701, SLE is 805921 and SRE 807381. This continues until finally 212.192.64.74 sends a correct ACK of 817601 in frame 3704. This whole situation repeats beginning with an incorrect ACK in frame 4065 and continuing until there is a correct ACK in frame 4082. Once again, the incorrect ACK is 864 bytes into a 1460-byte segment. Each time this happens, the situation resolves itself in well under a millisecond, so it does not seem to have any effect on throughput and there doesn't seem to be any interruption of the data flow. I can't explain this behavior. I think you've encountered a bug. Do these weird SACK blocks correspond with the client receiving corrupted data? answered 31 Jan '13, 13:26  Jim Aragon Thank you for attention! These weird SACK blocks indeed correspond with corrupted data. Every time we notice it - browser reports a failed object and may show corrupted picture. This happens on Win7, IE9, Chrome, CentOS, firefox... Issue seen on hardware PCs and VMs... Server is CentOS 6.3 x64 (have HW and VM one) (31 Jan '13, 21:19) v_paranoid |
Thank you, Jasper! Never read 2883 before... Just read in RFC 2018 about Data Receiver Reneging. So data receiver MAY not keep SACKed data in buffer. But must keep ACKed data. In our case it seems like receiver indeed drop SACKed data before passing it to browser. And browser shows partial content after timeout.
Worst situation is between 10g HW server and 10g clients. Perhaps our network is too fast? )
And I think that erroneous ACK is sent by LRO of NIC. In server trace we see Large segments (more than 1514) and these false ACKs actually ACKing Large segments.
When we force hosts to disable SACK, issue remains the same but wireshark traces look not so funny )
As I understand, segment MUST contain SACKs for all out-of-order contigous parts.
So given that we see SACKs for newer segment only - it means previous out-of-order data is no longer in buffer.
We do not see retransmissions on the middle switch trace - each segment was sent by server once. So it is not D-SACKs.
I was not aware of D-SACK. @Jasper is quite right, the first SACK block can have a number lower than the cumulative ACK. I spent last night going over RFC 2883, and my brain hurts from trying to follow all the examples. And I agree that the problem is packet 3688, and also packet 4065.