Inconsistent delayed ACK behavior

Hi, packet pros. I've got an interesting one that I can't explain. Maybe someone out there has seen this behavior before.

Environment:

• Bandwidth: 1Gbps LAN
• RTT: <1ms
• Hosts: Both Win2k8R2, bare metal
• Test: wget of a ~223MB file from a Tomcat server

Issue: When MTU is default 1500, performance is ~55Mbps. Lower MTU to 1350 gets ~300Mbps.

Analysis: There are 10x more delayed ACKs in the 1500 MTU test vs the 1350 MTU test.

Fix: Disable delayed ACK (TcpAckFrequency = 1) = ~500Mbps.

Digging deeper and question: I can't see why a 1500 MTU (1460 MSS) would trigger more delayed ACKs than 1350 MTU (1310 MSS). On top of that, the delayed ACKs aren't consistent when there's a single segment to be ACKed that should trigger a delayed ACK.

In fact, the send buffer size is 64k so that leads to an odd number of packets for both 1460 (45 segments) and 1310 (51 segments) MSSes. So there should likely be a delayed ACK at the end of each 64k chunk.

However, the number of segments acknowledged per ACK varies, it's not always 2. Sometimes it varies wildly e.g. 2, 3, 8 or more. Given that the number of ACKs it takes to acknowledge 64k varies depending on how many segments are ACKed per ACK, it would make sense that sometimes we end up with a single segment to be ACKed at the end of 64k and sometimes not. So it doesn't make sense to me that a MSS of 1310 would have consistently 10x fewer delayed ACKs.

Now, this isn't my issue, it was brought to me, so I won't share the pcaps unless allowed (he's cool with it, see below), but I'll include some screencaps. Also, note that I added two columns Segments ACKed and Buffer bytes. I added these in packet-tcp.c.

Segments ACKed: The number of segments acknowledged by this ACK

Buffer bytes: The number of bytes sent since the last PSH flag. This gives me an idea of the send buffer size or how much data the application is giving to TCP. With low latency, bytes in flight never gets very high so I wanted an easier way to see the buffer size.

Example of single segment to be ACKED with delayed ACK:

Example of single segment to be ACKED with no delay:

These are just two examples that are repeated many times through out the capture. There are 3573 64k chunks which means with an MSS of 1460 and an odd number of packets, there could be 3573 delayed ACKs but there are only 150. There are only 15 delayed ACKs in the 1310 MSS capture.

I can not explain this behavior.

pcaps:

• 1500 MTU
• 1350 MTU