This is a static archive of our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

client sends ACKS very late and gets RST

0

This is a dump captured at the client (this is a web traffic generator tool) in a customer test environment. We were told that the client (.42) is not sending ACKs and this results in the test getting terminated. I am not an expert on networking or trace analysis and am having a tough time figuring out if the answer to why the client is not sending an ACK is hidden somewhere in the dump. I also have these basic questions -

  1. why is the client sending dup ACK for packet 43 - when it has recvd 2 pkts (44,45) following this. Isn't the DUP ACK for an earlier pkt sent when the client misses packets ?
  2. The server sends this seq no (23985) in 68, but client continues to send the DUP ACK
  3. Is it possible for the client to send ACKS this late -- eg., 90 is ACK for 56 (RTT - 3.4s), 90 is ACK for 57 (RTT= 11s), 96 has RTT 28s, 108 has RTT of 156s! Doesnt the ACK timer expire in about 500ms ?
  4. In 115 and 116 (seq 43547), server starts retransmitting right away and sends a RST at 125 - where as in the other cases, the server did nt send an RST though the client send an ACK after > 100 s.

pcap is at -- https://drive.google.com/file/d/0B6WbY-hKwR3KSWpQTHF2eVRhQTg/edit?usp=sharing

PS. I just realized that I cant upload an image - how do I convert the pcap to an image to upload ? --Jai

asked 06 Dec '13, 07:49

Jai's gravatar image

Jai
1113
accept rate: 0%

edited 06 Dec '13, 10:32

This is a dump captured at the client

As there is no link to the capture file, please post the file somewhere (google drive, dropbox, cloudshark.org or mega.co.nz) and then add the link to your question.

(06 Dec '13, 08:03) Kurt Knochner ♦

One Answer:

2

"why is the client sending dup ACK for packet 43? " Because the client's TCP did NOT see packet #43. The TCP connection is using SACK option and looking at the SACK left and tight edges it is clear that many packets that are seen in your trace never made it to the client's TCP layer. All the black background packets in your trace are not seen by TCP.

The trace was taken close to (in front of) the client but not in the client itself? If so, it is the 'SuperMic Super Micro Computer, Inc. ' device is dropping packets on its inbound interface. the packets arrive in the same instant, only the first 20 packets made it to TCP, so there might be a queue length of 20 packets...


Correction: Looking further down there are indications that the LRO Large Receive Offload is in place. Segments arrive that larger than the negotiated MSS filter: tcp.len gt 1448 : . So the trace was obviously taken behind the NIC card and the packets are therefore not dropped there. Also the packet rate at the end of the trace is close to zero, certainly not overloading any interfaces there. So there must be something else discarding the packets within the client. The TCP checksumms on inbound packets are correct however.

My next action would be to check whether Large Receive Offload or Generic Receive Offload are enabled for the interface and turn those off: ethtool -k eth0 will show the current settings. alt text

answered 07 Dec '13, 03:52

mrEEde's gravatar image

mrEEde
3.9k152270
accept rate: 20%

edited 07 Dec '13, 23:01

Thanks very much for looking at the dump. Yes, you're right that the dump was captured on the client. LRO and GRO were turned on, so we ve tried turning off LRO and GRO as you suggested. It did appear to make a difference in the http errors initially, but when we turned it back on to confirm, we didn't see the errors again either. We're increasing the traffic to see if it reappears. If the pkts are dropped in the client (behind the NIC), how could this have changed ?

(11 Dec '13, 02:00) Jai

I have no idea. I don't know which TCP stack is running there, certainly not one that I would use to stress test a web server though.

(11 Dec '13, 11:35) mrEEde