This is a static archive of our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

Receiver limits TCP window size to 64,512

Facts (please identify any false statements):

I have a 100 Mbps connection between two sites that are 80 ms apart
This is a long fat connection that could benefit from a large TCP window size perhaps up to 100 Mbps * 0.08 sec = 1,000,000 bytes
Both machines are running Windows Server 2012. "Receive window auto tuning level" is normal on both. "Window scaling heuristics" are disabled on both.
I ran "iperf -s" on one side and "iperf -c" on the other. The transfer happened at 5 Mbps. I get the same result going the other direction.
Both sides advertised support for TCP sliding windows in their SYNs.
The receiver requested a TCP window size of 64,512 bytes (0xFC00) during the entire run with a TCP window scale value of "no shift" (0x000).
The network was able to handle a larger window size (see sequence diagrams below)
The receiver kept the window smaller than the network supports
This connection is happening within an IPSEC VPN. MTU of the tunnel interface is reduced to 1400 bytes in both directions.

Question

Why is the receiver keeping the window small?

Sender TCP Settings

PS C:\Users\acs> netsh interface tcp show global Querying active state... TCP Global Parameters Receive-Side Scaling State : enabled Chimney Offload State : disabled NetDMA State : disabled Direct Cache Access (DCA) : disabled Receive Window Auto-Tuning Level : normal Add-On Congestion Control Provider : none ECN Capability : enabled RFC 1323 Timestamps : disabled Initial RTO : 3000 Receive Segment Coalescing State : enabled PS C:\Users\acs> netsh interface tcp show heuristics TCP Window Scaling heuristics Parameters Window Scaling heuristics : disabled Qualifying Destination Threshold : 3 Profile type unknown : normal Profile type public : normal Profile type private : normal Profile type domain : normal PS C:\Users\acs> Get-NetTCPSetting SettingName : Automatic MinRto(ms) : InitialCongestionWindow(MSS) : CongestionProvider : CwndRestart : DelayedAckTimeout(ms) : MemoryPressureProtection : AutoTuningLevelLocal : AutoTuningLevelGroupPolicy : AutoTuningLevelEffective : EcnCapability : Timestamps : InitialRto(ms) : ScalingHeuristics : DynamicPortRangeStartPort : DynamicPortRangeNumberOfPorts : SettingName : Custom MinRto(ms) : 20 InitialCongestionWindow(MSS) : 4 CongestionProvider : DCTCP CwndRestart : True DelayedAckTimeout(ms) : 10 MemoryPressureProtection : Enabled AutoTuningLevelLocal : Normal AutoTuningLevelGroupPolicy : NotConfigured AutoTuningLevelEffective : Local EcnCapability : Enabled Timestamps : Disabled InitialRto(ms) : 3000 ScalingHeuristics : Disabled DynamicPortRangeStartPort : 49152 DynamicPortRangeNumberOfPorts : 16384 SettingName : Compat MinRto(ms) : 300 InitialCongestionWindow(MSS) : 2 CongestionProvider : Default CwndRestart : False DelayedAckTimeout(ms) : 200 MemoryPressureProtection : Enabled AutoTuningLevelLocal : Normal AutoTuningLevelGroupPolicy : NotConfigured AutoTuningLevelEffective : Local EcnCapability : Enabled Timestamps : Disabled InitialRto(ms) : 3000 ScalingHeuristics : Disabled DynamicPortRangeStartPort : 49152 DynamicPortRangeNumberOfPorts : 16384 SettingName : Datacenter MinRto(ms) : 20 InitialCongestionWindow(MSS) : 4 CongestionProvider : DCTCP CwndRestart : True DelayedAckTimeout(ms) : 10 MemoryPressureProtection : Enabled AutoTuningLevelLocal : Normal AutoTuningLevelGroupPolicy : NotConfigured AutoTuningLevelEffective : Local EcnCapability : Enabled Timestamps : Disabled InitialRto(ms) : 3000 ScalingHeuristics : Disabled DynamicPortRangeStartPort : 49152 DynamicPortRangeNumberOfPorts : 16384

SettingName : Internet MinRto(ms) : 300 InitialCongestionWindow(MSS) : 4 CongestionProvider : CTCP CwndRestart : False DelayedAckTimeout(ms) : 50 MemoryPressureProtection : Enabled AutoTuningLevelLocal : Normal AutoTuningLevelGroupPolicy : NotConfigured AutoTuningLevelEffective : Local EcnCapability : Enabled Timestamps : Disabled InitialRto(ms) : 3000 ScalingHeuristics : Disabled DynamicPortRangeStartPort : 49152 DynamicPortRangeNumberOfPorts : 16384

Sender SYN

No.     Time           Source                Destination           Protocol Length Delta      Sequence number Acknowledgment number Bytes in flight Calculated window size Info
814 5.036577000    10.10.0.21            10.11.0.1             TCP      66     0.000000000 0               0                                     64512                  49758→5001 [SYN, ECN, CWR] Seq=0 Win=64512 Len=0 MSS=1460 WS=1 SACK_PERM=1
Frame 814: 66 bytes on wire (528 bits), 66 bytes captured (528 bits) on interface 0
Ethernet II, Src: 00:11:22:33:44:55, Dst: aa:bb:cc:dd:ee:ff
Internet Protocol Version 4, Src: 10.10.0.21 (10.10.0.21), Dst: 10.11.0.1 (10.11.0.1)
Transmission Control Protocol, Src Port: 49758 (49758), Dst Port: 5001 (5001), Seq: 0, Len: 0
Source Port: 49758 (49758)
Destination Port: 5001 (5001)
[Stream index: 73]
[TCP Segment Len: 0]
Sequence number: 0    (relative sequence number)
Acknowledgment number: 0
Header Length: 32 bytes
…. 0000 1100 0010 = Flags: 0x0c2 (SYN, ECN, CWR)
Window size value: 64512
[Calculated window size: 64512]
Checksum: 0x1451 [validation disabled]
Urgent pointer: 0
Options: (12 bytes), Maximum segment size, No-Operation (NOP), Window scale, No-Operation (NOP), No-Operation (NOP), SACK permitted
Maximum segment size: 1460 bytes
No-Operation (NOP)
Window scale: 0 (multiply by 1)
Kind: Window Scale (3)
Length: 3
Shift count: 0
[Multiplier: 1]
No-Operation (NOP)
No-Operation (NOP)
TCP SACK Permitted Option: True

Sender perspective of sequence graph

Sequence graph

Receiver TCP Settings

PS C:\Users\acs> netsh interface tcp show global Querying active state… TCP Global Parameters Receive-Side Scaling State : enabled Chimney Offload State : disabled NetDMA State : disabled Direct Cache Access (DCA) : disabled Receive Window Auto-Tuning Level : normal Add-On Congestion Control Provider : none ECN Capability : enabled RFC 1323 Timestamps : disabled Initial RTO : 3000 Receive Segment Coalescing State : enabled PS C:\Users\acs> netsh interface tcp show heuristics TCP Window Scaling heuristics Parameters Window Scaling heuristics : disabled Qualifying Destination Threshold : 3 Profile type unknown : normal Profile type public : normal Profile type private : normal Profile type domain : normal PS C:\Users\acs> Get-NetTCPSetting SettingName : Automatic MinRto(ms) : InitialCongestionWindow(MSS) : CongestionProvider : CwndRestart : DelayedAckTimeout(ms) : MemoryPressureProtection : AutoTuningLevelLocal : AutoTuningLevelGroupPolicy : AutoTuningLevelEffective : EcnCapability : Timestamps : InitialRto(ms) : ScalingHeuristics : DynamicPortRangeStartPort : DynamicPortRangeNumberOfPorts : SettingName : Custom MinRto(ms) : 20 InitialCongestionWindow(MSS) : 4 CongestionProvider : DCTCP CwndRestart : True DelayedAckTimeout(ms) : 10 MemoryPressureProtection : Enabled AutoTuningLevelLocal : Normal AutoTuningLevelGroupPolicy : NotConfigured AutoTuningLevelEffective : Local EcnCapability : Enabled Timestamps : Disabled InitialRto(ms) : 3000 ScalingHeuristics : Disabled DynamicPortRangeStartPort : 49152 DynamicPortRangeNumberOfPorts : 16384 SettingName : Compat MinRto(ms) : 300 InitialCongestionWindow(MSS) : 2 CongestionProvider : Default CwndRestart : False DelayedAckTimeout(ms) : 200 MemoryPressureProtection : Enabled AutoTuningLevelLocal : Normal AutoTuningLevelGroupPolicy : NotConfigured AutoTuningLevelEffective : Local EcnCapability : Enabled Timestamps : Disabled InitialRto(ms) : 3000 ScalingHeuristics : Disabled DynamicPortRangeStartPort : 49152 DynamicPortRangeNumberOfPorts : 16384 SettingName : Datacenter MinRto(ms) : 20 InitialCongestionWindow(MSS) : 4 CongestionProvider : DCTCP CwndRestart : True DelayedAckTimeout(ms) : 10 MemoryPressureProtection : Enabled AutoTuningLevelLocal : Normal AutoTuningLevelGroupPolicy : NotConfigured AutoTuningLevelEffective : Local EcnCapability : Enabled Timestamps : Disabled InitialRto(ms) : 3000 ScalingHeuristics : Disabled DynamicPortRangeStartPort : 49152 DynamicPortRangeNumberOfPorts : 16384

Receiver SYN

No.     Time           Source                Destination           Protocol Length Delta      Sequence number Acknowledgment number Bytes in flight Calculated window size Info
817 5.110501000    10.11.0.1             10.10.0.21            TCP      70     0.073924000 0               1                                     64512                  5001→49758 [SYN, ACK, ECN] Seq=0 Ack=1 Win=64512 Len=0 MSS=1460 WS=1 SACK_PERM=1 [ETHERNET FRAME CHECK SEQUENCE INCORRECT]
Frame 817: 70 bytes on wire (560 bits), 70 bytes captured (560 bits) on interface 0
Ethernet II, Src: aa:bb:cc:dd:ee:ff, Dst: 00:11:22:33:44:55
Internet Protocol Version 4, Src: 10.11.0.1 (10.11.0.1), Dst: 10.10.0.21 (10.10.0.21)
Transmission Control Protocol, Src Port: 5001 (5001), Dst Port: 49758 (49758), Seq: 0, Ack: 1, Len: 0
Source Port: 5001 (5001)
Destination Port: 49758 (49758)
[Stream index: 73]
[TCP Segment Len: 0]
Sequence number: 0    (relative sequence number)
Acknowledgment number: 1    (relative ack number)
Header Length: 32 bytes
…. 0000 0101 0010 = Flags: 0x052 (SYN, ACK, ECN)
Window size value: 64512
[Calculated window size: 64512]
Checksum: 0xb5bb [validation disabled]
Urgent pointer: 0
Options: (12 bytes), Maximum segment size, No-Operation (NOP), Window scale, No-Operation (NOP), No-Operation (NOP), SACK permitted
Maximum segment size: 1460 bytes
No-Operation (NOP)
Window scale: 0 (multiply by 1)
Kind: Window Scale (3)
Length: 3
Shift count: 0
[Multiplier: 1]
No-Operation (NOP)
No-Operation (NOP)
TCP SACK Permitted Option: True
[SEQ/ACK analysis]

Receiver perspective of sequence graph

Sequence graph

tcpwindowsize

asked 12 Jun ‘15, 12:58

chrisstankevitz
6●1●1●2
accept rate: 0%

Have you ever tried to disable the autotuning level?

(12 Jun ‘15, 14:14) Christian_R

My question would be: did you ever see the window size being a problem? Meaning: did the trace show symptoms at any point in time that the sender had to actually wait for an ACK before it continue to send? You should see things like “Window Full” etc. if that happens, plus long delta times before continuing to send.

(12 Jun ‘15, 15:16) Jasper ♦♦

@Christian_R: No, my autotuning is enabled. Are you suggesting that disabling autotuning will “solve” my problem? Or are you concerned that I inadvertently disabled autotuning and that is why I am having a problem? I believe “disable autotuning” causes the receive window to be stuck at some fixed value, which is not what I’m after.

(15 Jun ‘15, 09:08) chrisstankevitz

@Jasper: Thank you. You are correct, no “window full”. I now believe the question is “When one is not specified, why isn’t Windows Server 2012 dynamically increasing SO_SNDBUF/SO_RCVBUF to accommodate long fat pipes as described at MSDN?"

(15 Jun ‘15, 09:12) chrisstankevitz

Wouldn’t setting the buffer sizes be an application issue?

(15 Jun ‘15, 09:44) grahamb ♦

@christiankevitz: Easily said I just were interested if you tried it. I never tried, because of the same reason as you.

I have a side question out of topic, just for interest the questions really have nothing to do with the window size: Do you use nlb? Do you have a lot of multicasts/broadcast at the servers? Are the systems physical or virtualized?

(15 Jun ‘15, 15:19) Christian_R

@grahamb: indeed setting buffer sizes is an application issue. In this particular case, the application is not setting a buffer size… in which case I’m expecting Windows to dynamically increase the buffer sizes as needed per the MSDN link posted above. But apparently that is not happening and I’m trying to figure out why.

(16 Jun ‘15, 01:14) chrisstankevitz

Did you have any more joy with this? I also have a window size limited to this value despite setting larger buffer sizes. While its not a problem, as @Jasper pointed out, with the window getting full, I do believe it can limit the maximum congestion window size, which may be more of an issue. (And we do often have to wait for ACKs before it continues to send)

(19 Jun ‘15, 06:27) glyn_walters

showing 5 of 8 show 3 more comments