This is a static archive of our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

Delay in HTTP Status Packet and First TCP Segment

0

Hi,

I need help... can't figure out what is the cause of a Web Application Slowness Issue.
Can't think of what else to check anymore.

Web Application Server in Site A.
Users in Site B & C tries to access the Web Application.
Site A, B & C uses Riverbed WANX, however the traffic for this application is "Passthrough Intentional".


Site A <------------Tunnel-----------> Central <------------Tunnel-----------> Site B
Site A <-Tunnel-> ISP <-Tunnel-> Central <-Tunnel-> ISP <-Tunnel-> Site B

Users from Site B, have no issues regardless of Client Operating System.


Site A <------------Tunnel-----------> Central <------------Tunnel-----------> Site C
Site A <-Tunnel-> ISP <-Tunnel-> Central <-Tunnel-> ISP <-Tunnel-> Site C

Users from Site C, experience slowness on Windows Vista and 7 only. No issues with Windows XP.


Initial assumptions:
- OS problem, since XP has no problem
- However, Site B has no issues regardless of Operating System.

Additional Notes:
- Both Site B & C uses the same model of Routers and Firewall
- Their Firmware version are the same as well
- Both site has MTU of 1500 (Tested using ping, result = 1472)
- The Vista and 7 Machines tested in Site B & C are the same (We move them over)
- The ISP for all sites are the same provider, going through a private MPLS cloud

Troubleshooting tried, but doesn't work
- Disabled Autotune on Windows Vista and 7
- Site A ping to Application Server: 100 % (10000/10000), round-trip min/avg/max = 1/1/21 ms
- Site A ping to Site B: 100 % (10000/10000), round-trip min/avg/max = 2/2/23 ms
- Site A ping to Site C: 100 % (10000/10000), round-trip min/avg/max = 2/2/11 ms
- Access application at night where there are no congestion from Site C, results = slow

Captured traffic from Site C and Site A, using "Time since previous frame" to check delays:
- Segmented TCP "HTTP GET" has no delay when it leaves Site C (User Site)
- Viewing from Site A (Server Site), 5 sec delay is found ONLY in the first frame of the Segmented TCP "HTTP GET"
- When Segmented TCP "HTTP/1.1 OK" is returned, it has no delay when it leaves Site A (Server Site)
- However viewing from Site C (User Site), 5 sec delay is found "ONLY" in the first frame of "ALL" Segmented TCP "HTTP/1.1 OK"
- Additionally "HTTP/1.1 304 Not Modified" messages that are "NOT" segmented has a 5 seconds delay as well


alt text alt text alt text alt text

asked 12 Dec '13, 11:17

RebirthX's gravatar image

RebirthX
1●1●2●2
accept rate: 0%

edited 12 Dec '13, 12:15

You wrote: Both site has MTU of 1500 (Tested using ping, result = 1472)

Did you specify the -f option on the ping command? What is the largest size that gets through?

"-f : Specifies that Echo Request messages are sent with the Don't Fragment flag in the IP header set to 1"

(13 Dec '13, 23:04) mrEEde

Has this ever performed well and suddenly started to slow down on Site C?

(13 Dec '13, 23:06) mrEEde

Yes.
I did ping -f -l 1472 [IP Addr.]
1473 will results in packet needs to be fragmented.

(14 Dec '13, 02:58) RebirthX

They only realised this when the APP Users recently changed from WinXP to Win7.

(14 Dec '13, 03:00) RebirthX

Hmm, still very mysterious... What is the largest segment leaving the WinXP client? What is the largest segment leaving the Win7 client? Is the ip.flags.df bit set? Do you see any retransmissions occuring in the client traces?

At the server, what is the largest segment that arrives? Is the IP don't fragment bit set when those packets arrive? What is the TTL value?

(14 Dec '13, 23:02) mrEEde

Win XP Largest Segment Leave = 536
Win7 Largest Segment Leave = 536
Server Largest Segment Arrive = 536

Win XP's IP DF Flag = 0x02
Win7's IP DF Flag = 0x02

They seems to be all the same, but only Win7 & Vista is slow.

By the way, I've tried accessing another Web server in the same subnet.
The Largest Segment is 1460.
Does that means that the TCP MSS might be a configuration issue in the Server itself?
It is a Win Server 2003 virtualized in a ESXi by the way.

(15 Dec '13, 01:43) RebirthX
showing 5 of 6 show 1 more comments

3 Answers:

0

The segment size of 536 indicates that you are effectively using the default MTU size of 576. Did you check the SYN and SYN_ACK packet what the negotiated MSS was in the 3-way handshake? If this is a normal offering - 1380 or 1460 then you are suffering from - failing - Path MTU discovery where - if all else fails - windows falls back to 576...

Did you see any ICMP packets at the client / server side indicating that fragmentation is required?

Additional Information: The 3-way handshakes indicate the both clients offer a MSS of 1260 on their outbound SYN request. The SYN_ACK from the server shows a MSS of 1460 bytes in both cases.

As the infrastructure consists of 'tunnels' it is good practice in the network to adjust the MSS values in the SYN packets as they are entering a VPN tunnel to avoid fragmentation problems. This does not happen. Please contact your ISPs and ask them to tcp-adjust-mss to avoid the PMTUD logic which does not seem to work properly as we see 536-bytes segments at the server lateron in the flow.

answered 12 Dec '13, 14:45

mrEEde's gravatar image

mrEEde
3.9k●15●22●70
accept rate: 20%

edited 12 Dec '13, 23:08

Hmm... Yeah...
I didn't know that the Segmentation size should be around the MTU size...
That may be a problem.

But I don't think this is the cause of the problem actually.
Cause the XP Machine's Segmentation is also 536, but the time it takes to load the Web Link is almost instance.
While the Vista/7 Machine takes about a minute...
I do not know how to analyse the MSS in the 3-way, can you tell me what to see?

The following is the TCP SYN/ACK of the XP

No.     Time           Source                Destination           Protocol Length Delay      Window size scaling factor Info
     10 0.060064000    XP Client IP           Server IP             TCP      62     0.000000000                            1679 > 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1260 SACK_PERM=1
Frame 10: 62 bytes on wire (496 bits), 62 bytes captured (496 bits) on interface 0
Ethernet II, Src: , Dst: 
Internet Protocol Version 4, Src: XP Client IP IP, Dst: Server IP
Transmission Control Protocol, Src Port: 1679 (1679), Dst Port: 80 (80), Seq: 0, Len: 0
    Source port: 1679 (1679)
    Destination port: 80 (80)
    [Stream index: 1]
    Sequence number: 0    (relative sequence number)
    Header length: 28 bytes
    Flags: 0x002 (SYN)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...0 .... = Acknowledgment: Not set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish request (SYN): server port 80]
                [Message: Connection establish request (SYN): server port 80]
                [Severity level: Chat]
                [Group: Sequence]
        .... .... ...0 = Fin: Not set
    Window size value: 65535
    [Calculated window size: 65535]
    Checksum: 0x016d [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (8 bytes), Maximum segment size, No-Operation (NOP), No-Operation (NOP), SACK permitted
        Maximum segment size: 1260 bytes
            Kind: MSS size (2)
            Length: 4
            MSS Value: 1260
        No-Operation (NOP)
            Type: 1
                0... .... = Copy on fragmentation: No
                .00. .... = Class: Control (0)
                ...0 0001 = Number: No-Operation (NOP) (1)
        No-Operation (NOP)
            Type: 1
                0... .... = Copy on fragmentation: No
                .00. .... = Class: Control (0)
                ...0 0001 = Number: No-Operation (NOP) (1)
        TCP SACK Permitted Option: True
            Kind: SACK Permission (4)
            Length: 2
    [Timestamps]
        [Time since first frame in this TCP stream: 0.000000000 seconds]
        [Time since previous frame in this TCP stream: 0.000000000 seconds]
No.     Time           Source                Destination           Protocol Length Delay      Window size scaling factor Info
     11 0.065313000    Server IP             XP Client IP           TCP      62     0.005249000                            80 > 1679 [SYN, ACK] Seq=0 Ack=1 Win=64240 Len=0 MSS=1460 SACK_PERM=1
Frame 11: 62 bytes on wire (496 bits), 62 bytes captured (496 bits) on interface 0
Ethernet II, Src: Alcatel-_39:43:80 (e8:e7:32:39:43:80), Dst: 
Internet Protocol Version 4, Src: Server IP, Dst: XP Client IP
Transmission Control Protocol, Src Port: 80 (80), Dst Port: 1679 (1679), Seq: 0, Ack: 1, Len: 0
    Source port: 80 (80)
    Destination port: 1679 (1679)
    [Stream index: 1]
    Sequence number: 0    (relative sequence number)
    Acknowledgment number: 1    (relative ack number)
    Header length: 28 bytes
    Flags: 0x012 (SYN, ACK)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...1 .... = Acknowledgment: Set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish acknowledge (SYN+ACK): server port 80]
                [Message: Connection establish acknowledge (SYN+ACK): server port 80]
                [Severity level: Chat]
                [Group: Sequence]
        .... .... ...0 = Fin: Not set
    Window size value: 64240
    [Calculated window size: 64240]
    Checksum: 0x8a5a [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (8 bytes), Maximum segment size, No-Operation (NOP), No-Operation (NOP), SACK permitted
        Maximum segment size: 1460 bytes
            Kind: MSS size (2)
            Length: 4
            MSS Value: 1460
        No-Operation (NOP)
            Type: 1
                0... .... = Copy on fragmentation: No
                .00. .... = Class: Control (0)
                ...0 0001 = Number: No-Operation (NOP) (1)
        No-Operation (NOP)
            Type: 1
                0... .... = Copy on fragmentation: No
                .00. .... = Class: Control (0)
                ...0 0001 = Number: No-Operation (NOP) (1)
        TCP SACK Permitted Option: True
            Kind: SACK Permission (4)
            Length: 2
    [SEQ/ACK analysis]
    [Timestamps]
        [Time since first frame in this TCP stream: 0.005249000 seconds]
        [Time since previous frame in this TCP stream: 0.005249000 seconds]
No.     Time           Source                Destination           Protocol Length Delay      Window size scaling factor Info
     12 0.068621000    XP Client IP           Server IP             TCP      54     0.003308000 -2                         1679 > 80 [ACK] Seq=1 Ack=1 Win=65535 Len=0
Frame 12: 54 bytes on wire (432 bits), 54 bytes captured (432 bits) on interface 0
Ethernet II, Src: , Dst: 
Internet Protocol Version 4, Src: XP Client IP, Dst: Server IP
Transmission Control Protocol, Src Port: 1679 (1679), Dst Port: 80 (80), Seq: 1, Ack: 1, Len: 0
    Source port: 1679 (1679)
    Destination port: 80 (80)
    [Stream index: 1]
    Sequence number: 1    (relative sequence number)
    Acknowledgment number: 1    (relative ack number)
    Header length: 20 bytes
    Flags: 0x010 (ACK)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...1 .... = Acknowledgment: Set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..0. = Syn: Not set
        .... .... ...0 = Fin: Not set
    Window size value: 65535
    [Calculated window size: 65535]
    [Window size scaling factor: -2 (no window scaling used)]
    Checksum: 0xb20f [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    [SEQ/ACK analysis]
    [Timestamps]
        [Time since first frame in this TCP stream: 0.008557000 seconds]
        [Time since previous frame in this TCP stream: 0.003308000 seconds]



The following is the TCP SYN/ACK the Win7 Machine

No.     Time        Source                Destination           Protocol Length Delay      Window size scaling factor Info
      1 0.000000    Win7 Client IP          Server IP             TCP      66     0.000000000                            49595 > 80 [SYN] Seq=0 Win=8192 Len=0 MSS=1260 WS=4 SACK_PERM=1
Frame 1: 66 bytes on wire (528 bits), 66 bytes captured (528 bits)
Ethernet II, Src: , Dst: 
Internet Protocol Version 4, Src: Win7 Client IP, Dst: Server IP
Transmission Control Protocol, Src Port: 49595 (49595), Dst Port: 80 (80), Seq: 0, Len: 0
    Source port: 49595 (49595)
    Destination port: 80 (80)
    [Stream index: 0]
    Sequence number: 0    (relative sequence number)
    Header length: 32 bytes
    Flags: 0x002 (SYN)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...0 .... = Acknowledgment: Not set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish request (SYN): server port 80]
                [Message: Connection establish request (SYN): server port 80]
                [Severity level: Chat]
                [Group: Sequence]
        .... .... ...0 = Fin: Not set
    Window size value: 8192
    [Calculated window size: 8192]
    Checksum: 0x10d2 [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (12 bytes), Maximum segment size, No-Operation (NOP), Window scale, No-Operation (NOP), No-Operation (NOP), SACK permitted
    [Timestamps]
        [Time since first frame in this TCP stream: 0.000000000 seconds]
        [Time since previous frame in this TCP stream: 0.000000000 seconds]
No.     Time        Source                Destination           Protocol Length Delay      Window size scaling factor Info
      2 0.002502    Server IP             Win7 Client IP          TCP      66     0.002502000                            80 > 49595 [SYN, ACK] Seq=0 Ack=1 Win=64240 Len=0 MSS=1460 WS=1 SACK_PERM=1
Frame 2: 66 bytes on wire (528 bits), 66 bytes captured (528 bits)
Ethernet II, Src: Cisco_c3:af:c3 (00:25:83:c3:af:c3), Dst: 
Internet Protocol Version 4, Src: Server IP, Dst: Win7 Client IP
Transmission Control Protocol, Src Port: 80 (80), Dst Port: 49595 (49595), Seq: 0, Ack: 1, Len: 0
    Source port: 80 (80)
    Destination port: 49595 (49595)
    [Stream index: 0]
    Sequence number: 0    (relative sequence number)
    Acknowledgment number: 1    (relative ack number)
    Header length: 32 bytes
    Flags: 0x012 (SYN, ACK)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...1 .... = Acknowledgment: Set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish acknowledge (SYN+ACK): server port 80]
                [Message: Connection establish acknowledge (SYN+ACK): server port 80]
                [Severity level: Chat]
                [Group: Sequence]
        .... .... ...0 = Fin: Not set
    Window size value: 64240
    [Calculated window size: 64240]
    Checksum: 0x1882 [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (12 bytes), Maximum segment size, No-Operation (NOP), Window scale, No-Operation (NOP), No-Operation (NOP), SACK permitted
    [SEQ/ACK analysis]
        [This is an ACK to the segment in frame: 1]
        [The RTT to ACK the segment was: 0.002502000 seconds]
    [Timestamps]
        [Time since first frame in this TCP stream: 0.002502000 seconds]
        [Time since previous frame in this TCP stream: 0.002502000 seconds]
No.     Time        Source                Destination           Protocol Length Delay      Window size scaling factor Info
      3 0.003231    Win7 Client IP          Server IP             TCP      60     0.000729000 4                          49595 > 80 [ACK] Seq=1 Ack=1 Win=66780 Len=0
Frame 3: 60 bytes on wire (480 bits), 60 bytes captured (480 bits)
Ethernet II, Src: , Dst: 
Internet Protocol Version 4, Src: Win7 Client IP, Dst: Server IP
Transmission Control Protocol, Src Port: 49595 (49595), Dst Port: 80 (80), Seq: 1, Ack: 1, Len: 0
    Source port: 49595 (49595)
    Destination port: 80 (80)
    [Stream index: 0]
    Sequence number: 1    (relative sequence number)
    Acknowledgment number: 1    (relative ack number)
    Header length: 20 bytes
    Flags: 0x010 (ACK)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...1 .... = Acknowledgment: Set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..0. = Syn: Not set
        .... .... ...0 = Fin: Not set
    Window size value: 16695
    [Calculated window size: 66780]
    [Window size scaling factor: 4]
    Checksum: 0x1307 [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    [SEQ/ACK analysis]
        [This is an ACK to the segment in frame: 2]
        [The RTT to ACK the segment was: 0.000729000 seconds]
    [Timestamps]
        [Time since first frame in this TCP stream: 0.003231000 seconds]
        [Time since previous frame in this TCP stream: 0.000729000 seconds]

(12 Dec '13, 21:35) RebirthX

0

Site A, B & C uses Riverbed WANX, however the traffic for this application is "Passthrough Intentional".

Well, maybe the traffic is not so 'passthrough' as you expect it to be due to the config option ;-))

I suggest to capture at several places in parallel to rule out some things. Please sync the clocks of the capture devices.

[SiteC]: client --- WANX -- Router ---- MPLS --- Router --- WANX ---- Server [SiteA]
                   |       |                               |         |
Capture:        here[1]  here[2]                          here[3]  here[4]

Start with a capture at [1] and [4] in parallel. Then compare the timestamps of the capture files. If you see the delay in [1] but not in [4], it's not the server, but something in the data path.

Continue with: [2] and [3] in parallel. Then compare the timestamps of the capture files. If you see the delay in [2] but not in [3] it is caused by the MPLS and you can hand over the problem to the ISP :-)

Continue with: [1] and [3] in parallel. Then compare the timestamps of the capture files. If you see the delay in [1] but not in [3], the problem is WANX[SiteC]. If you see the delay at [3] the problem is WANX[SiteA].

That's just a rough idea how to find the problem and I did not think through all possible combinations. I leave that up to you ;-))

Regards
Kurt

answered 13 Dec '13, 04:52

Kurt%20Knochner's gravatar image

Kurt Knochner ♦
24.8k●10●39●237
accept rate: 15%

0

Hi all,

Thanks for all the help, I've managed to resolved the problem.
I did captures from all points and found out the delay is caused by Site A's External Firewall.

Site A's External Firewall is using McAfee Enterprise Firewall 8.
Aside from Source/Destination IP and Port, there is this additional thing called "Application Defense Group".
I'm not sure what it does actually,but once it is disabled, the problem got resolved.
No more delays.

The reason why it does not affect Site B but only Site C, is because Site B & C have different Firewall Rules in Site A's Firewall.
The rule for Site C, has the "Application Defense Group" turned on.
The rule for Site B, has the "Application Defense Group" turned off.
Why does it not affect Windows XP but only affect Windows Vista & 7, I still have no idea.

alt text


As for the TCP MSS, which is a separate issue, is not fixed yet.
Accessing the Server from Site A's Core Switch, the TCP Segment size shows 536.
This will pass through the Server's Firewall in Site A, which is a Palo Alto.

We tried using another Server which is in the same subnet as the Application Server and did a capture, bypassing the Server Firewall.
The TCP Segment size shows 1460...
Seems like it only becomes 536 when it passes though the Palo Alto Firewall.
So... is this the Firewall problem or the Server's TCP MSS registry problem?

answered 16 Dec '13, 05:08

RebirthX's gravatar image

RebirthX
1●1●2●2
accept rate: 0%

edited 16 Dec '13, 05:12