This is our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

Facts (please identify any false statements):

  1. I have a 100 Mbps connection between two sites that are 80 ms apart

  2. This is a long fat connection that could benefit from a large TCP window size perhaps up to 100 Mbps * 0.08 sec = 1,000,000 bytes

  3. Both machines are running Windows Server 2012. "Receive window auto tuning level" is normal on both. "Window scaling heuristics" are disabled on both.

  4. I ran "iperf -s" on one side and "iperf -c" on the other. The transfer happened at 5 Mbps. I get the same result going the other direction.

  5. Both sides advertised support for TCP sliding windows in their SYNs.

  6. The receiver requested a TCP window size of 64,512 bytes (0xFC00) during the entire run with a TCP window scale value of "no shift" (0x000).

  7. The network was able to handle a larger window size (see sequence diagrams below)

  8. The receiver kept the window smaller than the network supports

  9. This connection is happening within an IPSEC VPN. MTU of the tunnel interface is reduced to 1400 bytes in both directions.

Question

Why is the receiver keeping the window small?


Sender TCP Settings

PS C:\Users\acs> netsh interface tcp show global
Querying active state...

TCP Global Parameters
----------------------------------------------
Receive-Side Scaling State          : enabled
Chimney Offload State               : disabled
NetDMA State                        : disabled
Direct Cache Access (DCA)           : disabled
Receive Window Auto-Tuning Level    : normal
Add-On Congestion Control Provider  : none
ECN Capability                      : enabled
RFC 1323 Timestamps                 : disabled
Initial RTO                         : 3000
Receive Segment Coalescing State    : enabled

PS C:\Users\acs> netsh interface tcp show heuristics
TCP Window Scaling heuristics Parameters
----------------------------------------------
Window Scaling heuristics         : disabled
Qualifying Destination Threshold  : 3
Profile type unknown              : normal
Profile type public               : normal
Profile type private              : normal
Profile type domain               : normal

PS C:\Users\acs> Get-NetTCPSetting

SettingName                   : Automatic
MinRto(ms)                    : 
InitialCongestionWindow(MSS)  : 
CongestionProvider            : 
CwndRestart                   : 
DelayedAckTimeout(ms)         : 
MemoryPressureProtection      : 
AutoTuningLevelLocal          : 
AutoTuningLevelGroupPolicy    : 
AutoTuningLevelEffective      : 
EcnCapability                 : 
Timestamps                    : 
InitialRto(ms)                : 
ScalingHeuristics             : 
DynamicPortRangeStartPort     : 
DynamicPortRangeNumberOfPorts :

SettingName                   : Custom
MinRto(ms)                    : 20
InitialCongestionWindow(MSS)  : 4
CongestionProvider            : DCTCP
CwndRestart                   : True
DelayedAckTimeout(ms)         : 10
MemoryPressureProtection      : Enabled
AutoTuningLevelLocal          : Normal
AutoTuningLevelGroupPolicy    : NotConfigured
AutoTuningLevelEffective      : Local
EcnCapability                 : Enabled
Timestamps                    : Disabled
InitialRto(ms)                : 3000
ScalingHeuristics             : Disabled
DynamicPortRangeStartPort     : 49152
DynamicPortRangeNumberOfPorts : 16384

SettingName                   : Compat
MinRto(ms)                    : 300
InitialCongestionWindow(MSS)  : 2
CongestionProvider            : Default
CwndRestart                   : False
DelayedAckTimeout(ms)         : 200
MemoryPressureProtection      : Enabled
AutoTuningLevelLocal          : Normal
AutoTuningLevelGroupPolicy    : NotConfigured
AutoTuningLevelEffective      : Local
EcnCapability                 : Enabled
Timestamps                    : Disabled
InitialRto(ms)                : 3000
ScalingHeuristics             : Disabled
DynamicPortRangeStartPort     : 49152
DynamicPortRangeNumberOfPorts : 16384

SettingName                   : Datacenter
MinRto(ms)                    : 20
InitialCongestionWindow(MSS)  : 4
CongestionProvider            : DCTCP
CwndRestart                   : True
DelayedAckTimeout(ms)         : 10
MemoryPressureProtection      : Enabled
AutoTuningLevelLocal          : Normal
AutoTuningLevelGroupPolicy    : NotConfigured
AutoTuningLevelEffective      : Local
EcnCapability                 : Enabled
Timestamps                    : Disabled
InitialRto(ms)                : 3000
ScalingHeuristics             : Disabled
DynamicPortRangeStartPort     : 49152
DynamicPortRangeNumberOfPorts : 16384

SettingName                   : Internet
MinRto(ms)                    : 300
InitialCongestionWindow(MSS)  : 4
CongestionProvider            : CTCP
CwndRestart                   : False
DelayedAckTimeout(ms)         : 50
MemoryPressureProtection      : Enabled
AutoTuningLevelLocal          : Normal
AutoTuningLevelGroupPolicy    : NotConfigured
AutoTuningLevelEffective      : Local
EcnCapability                 : Enabled
Timestamps                    : Disabled
InitialRto(ms)                : 3000
ScalingHeuristics             : Disabled
DynamicPortRangeStartPort     : 49152
DynamicPortRangeNumberOfPorts : 16384

Sender SYN

No.     Time           Source                Destination           Protocol Length Delta      Sequence number Acknowledgment number Bytes in flight Calculated window size Info
    814 5.036577000    10.10.0.21            10.11.0.1             TCP      66     0.000000000 0               0                                     64512                  49758→5001 [SYN, ECN, CWR] Seq=0 Win=64512 Len=0 MSS=1460 WS=1 SACK_PERM=1

Frame 814: 66 bytes on wire (528 bits), 66 bytes captured (528 bits) on interface 0
Ethernet II, Src: 00:11:22:33:44:55, Dst: aa:bb:cc:dd:ee:ff
Internet Protocol Version 4, Src: 10.10.0.21 (10.10.0.21), Dst: 10.11.0.1 (10.11.0.1)
Transmission Control Protocol, Src Port: 49758 (49758), Dst Port: 5001 (5001), Seq: 0, Len: 0
    Source Port: 49758 (49758)
    Destination Port: 5001 (5001)
    [Stream index: 73]
    [TCP Segment Len: 0]
    Sequence number: 0    (relative sequence number)
    Acknowledgment number: 0
    Header Length: 32 bytes
    .... 0000 1100 0010 = Flags: 0x0c2 (SYN, ECN, CWR)
    Window size value: 64512
    [Calculated window size: 64512]
    Checksum: 0x1451 [validation disabled]
    Urgent pointer: 0
    Options: (12 bytes), Maximum segment size, No-Operation (NOP), Window scale, No-Operation (NOP), No-Operation (NOP), SACK permitted
        Maximum segment size: 1460 bytes
        No-Operation (NOP)
        Window scale: 0 (multiply by 1)
            Kind: Window Scale (3)
            Length: 3
            Shift count: 0
            [Multiplier: 1]
        No-Operation (NOP)
        No-Operation (NOP)
        TCP SACK Permitted Option: True

Sender perspective of sequence graph

Sequence graph


Receiver TCP Settings

PS C:\Users\acs> netsh interface tcp show global
Querying active state...

TCP Global Parameters
----------------------------------------------
Receive-Side Scaling State          : enabled
Chimney Offload State               : disabled
NetDMA State                        : disabled
Direct Cache Access (DCA)           : disabled
Receive Window Auto-Tuning Level    : normal
Add-On Congestion Control Provider  : none
ECN Capability                      : enabled
RFC 1323 Timestamps                 : disabled
Initial RTO                         : 3000
Receive Segment Coalescing State    : enabled

PS C:\Users\acs> netsh interface tcp show heuristics
TCP Window Scaling heuristics Parameters
----------------------------------------------
Window Scaling heuristics         : disabled
Qualifying Destination Threshold  : 3
Profile type unknown              : normal
Profile type public               : normal
Profile type private              : normal
Profile type domain               : normal

PS C:\Users\acs> Get-NetTCPSetting

SettingName                   : Automatic
MinRto(ms)                    : 
InitialCongestionWindow(MSS)  : 
CongestionProvider            : 
CwndRestart                   : 
DelayedAckTimeout(ms)         : 
MemoryPressureProtection      : 
AutoTuningLevelLocal          : 
AutoTuningLevelGroupPolicy    : 
AutoTuningLevelEffective      : 
EcnCapability                 : 
Timestamps                    : 
InitialRto(ms)                : 
ScalingHeuristics             : 
DynamicPortRangeStartPort     : 
DynamicPortRangeNumberOfPorts :

SettingName                   : Custom
MinRto(ms)                    : 20
InitialCongestionWindow(MSS)  : 4
CongestionProvider            : DCTCP
CwndRestart                   : True
DelayedAckTimeout(ms)         : 10
MemoryPressureProtection      : Enabled
AutoTuningLevelLocal          : Normal
AutoTuningLevelGroupPolicy    : NotConfigured
AutoTuningLevelEffective      : Local
EcnCapability                 : Enabled
Timestamps                    : Disabled
InitialRto(ms)                : 3000
ScalingHeuristics             : Disabled
DynamicPortRangeStartPort     : 49152
DynamicPortRangeNumberOfPorts : 16384

SettingName                   : Compat
MinRto(ms)                    : 300
InitialCongestionWindow(MSS)  : 2
CongestionProvider            : Default
CwndRestart                   : False
DelayedAckTimeout(ms)         : 200
MemoryPressureProtection      : Enabled
AutoTuningLevelLocal          : Normal
AutoTuningLevelGroupPolicy    : NotConfigured
AutoTuningLevelEffective      : Local
EcnCapability                 : Enabled
Timestamps                    : Disabled
InitialRto(ms)                : 3000
ScalingHeuristics             : Disabled
DynamicPortRangeStartPort     : 49152
DynamicPortRangeNumberOfPorts : 16384

SettingName                   : Datacenter
MinRto(ms)                    : 20
InitialCongestionWindow(MSS)  : 4
CongestionProvider            : DCTCP
CwndRestart                   : True
DelayedAckTimeout(ms)         : 10
MemoryPressureProtection      : Enabled
AutoTuningLevelLocal          : Normal
AutoTuningLevelGroupPolicy    : NotConfigured
AutoTuningLevelEffective      : Local
EcnCapability                 : Enabled
Timestamps                    : Disabled
InitialRto(ms)                : 3000
ScalingHeuristics             : Disabled
DynamicPortRangeStartPort     : 49152
DynamicPortRangeNumberOfPorts : 16384

SettingName                   : Internet
MinRto(ms)                    : 300
InitialCongestionWindow(MSS)  : 4
CongestionProvider            : CTCP
CwndRestart                   : False
DelayedAckTimeout(ms)         : 50
MemoryPressureProtection      : Enabled
AutoTuningLevelLocal          : Normal
AutoTuningLevelGroupPolicy    : NotConfigured
AutoTuningLevelEffective      : Local
EcnCapability                 : Enabled
Timestamps                    : Disabled
InitialRto(ms)                : 3000
ScalingHeuristics             : Disabled
DynamicPortRangeStartPort     : 49152
DynamicPortRangeNumberOfPorts : 16384

Receiver SYN

No.     Time           Source                Destination           Protocol Length Delta      Sequence number Acknowledgment number Bytes in flight Calculated window size Info
    817 5.110501000    10.11.0.1             10.10.0.21            TCP      70     0.073924000 0               1                                     64512                  5001→49758 [SYN, ACK, ECN] Seq=0 Ack=1 Win=64512 Len=0 MSS=1460 WS=1 SACK_PERM=1 [ETHERNET FRAME CHECK SEQUENCE INCORRECT]

Frame 817: 70 bytes on wire (560 bits), 70 bytes captured (560 bits) on interface 0
Ethernet II, Src: aa:bb:cc:dd:ee:ff, Dst: 00:11:22:33:44:55
Internet Protocol Version 4, Src: 10.11.0.1 (10.11.0.1), Dst: 10.10.0.21 (10.10.0.21)
Transmission Control Protocol, Src Port: 5001 (5001), Dst Port: 49758 (49758), Seq: 0, Ack: 1, Len: 0
    Source Port: 5001 (5001)
    Destination Port: 49758 (49758)
    [Stream index: 73]
    [TCP Segment Len: 0]
    Sequence number: 0    (relative sequence number)
    Acknowledgment number: 1    (relative ack number)
    Header Length: 32 bytes
    .... 0000 0101 0010 = Flags: 0x052 (SYN, ACK, ECN)
    Window size value: 64512
    [Calculated window size: 64512]
    Checksum: 0xb5bb [validation disabled]
    Urgent pointer: 0
    Options: (12 bytes), Maximum segment size, No-Operation (NOP), Window scale, No-Operation (NOP), No-Operation (NOP), SACK permitted
        Maximum segment size: 1460 bytes
        No-Operation (NOP)
        Window scale: 0 (multiply by 1)
            Kind: Window Scale (3)
            Length: 3
            Shift count: 0
            [Multiplier: 1]
        No-Operation (NOP)
        No-Operation (NOP)
        TCP SACK Permitted Option: True
    [SEQ/ACK analysis]

Receiver perspective of sequence graph

Sequence graph

asked 12 Jun '15, 12:58

chrisstankevitz's gravatar image

chrisstankevitz
6112
accept rate: 0%

Have you ever tried to disable the autotuning level?

(12 Jun '15, 14:14) Christian_R

My question would be: did you ever see the window size being a problem? Meaning: did the trace show symptoms at any point in time that the sender had to actually wait for an ACK before it continue to send? You should see things like "Window Full" etc. if that happens, plus long delta times before continuing to send.

(12 Jun '15, 15:16) Jasper ♦♦

@Christian_R: No, my autotuning is enabled. Are you suggesting that disabling autotuning will "solve" my problem? Or are you concerned that I inadvertently disabled autotuning and that is why I am having a problem? I believe "disable autotuning" causes the receive window to be stuck at some fixed value, which is not what I'm after.

(15 Jun '15, 09:08) chrisstankevitz

@Jasper: Thank you. You are correct, no "window full". I now believe the question is "When one is not specified, why isn't Windows Server 2012 dynamically increasing SO_SNDBUF/SO_RCVBUF to accommodate long fat pipes as described at MSDN?"

(15 Jun '15, 09:12) chrisstankevitz

Wouldn't setting the buffer sizes be an application issue?

(15 Jun '15, 09:44) grahamb ♦

@christiankevitz: Easily said I just were interested if you tried it. I never tried, because of the same reason as you.

I have a side question out of topic, just for interest the questions really have nothing to do with the window size: Do you use nlb? Do you have a lot of multicasts/broadcast at the servers? Are the systems physical or virtualized?

(15 Jun '15, 15:19) Christian_R

@grahamb: indeed setting buffer sizes is an application issue. In this particular case, the application is not setting a buffer size... in which case I'm expecting Windows to dynamically increase the buffer sizes as needed per the MSDN link posted above. But apparently that is not happening and I'm trying to figure out why.

(16 Jun '15, 01:14) chrisstankevitz

Did you have any more joy with this? I also have a window size limited to this value despite setting larger buffer sizes. While its not a problem, as @Jasper pointed out, with the window getting full, I do believe it can limit the maximum congestion window size, which may be more of an issue. (And we do often have to wait for ACKs before it continues to send)

(19 Jun '15, 06:27) glyn_walters
showing 5 of 8 show 3 more comments
Be the first one to answer this question!
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×28

question asked: 12 Jun '15, 12:58

question was seen: 2,591 times

last updated: 19 Jun '15, 06:27

p​o​w​e​r​e​d by O​S​Q​A