Hi All i am trying to figure out as to why the client after receiving 302 is delaying to close the connection and initiate new one for the url in Location header. Basically the traffic to the Web App (via load balancer is as follows): 1) the client posts with a HTTP POST containing their login details 2) the server accepts that message, commits it, and responds to the client by sending a HTTP 302 redirect to a page called .... 3) The client when receiving the 302 would close its TCP session, and reestablish a new one and send a HTTP GET for the new URL provided in the 302 redirect. 4) Client confirms receiving 302 and : The main question is as to why after client’s receiving 302 is not terminating existing connection …? Any ideas ? asked 07 Aug ‘12, 15:21  andrus |
2 Answers:
well, hard to say without any information about the client beeing used... Please add this:
Regards answered 07 Aug '12, 17:00  Kurt Knochner ♦ edited 07 Aug '12, 17:00 Thanks for checking this Kurt, Here is more info for this: - OS is Windows 7 - Browser is Firefox 14 (has been tested using IE 8 - There is no Javascript in the page code The flow is a follows: (Client,win7) <-> (Load balancer) <-> (backend IIS,win2008) have traces from all 3 machines, I narrowed it down to the issue with the client not responding to 302 in timely fashion manner NOTE: both traces were taken using the same client machine and same browser, the change is that in Working scenario we have older version of firmware on Load balancer and in the SLOW working scenario we has newest version Here is the properly working trace:( clients IP 192.168.1.37) and here is the delay in clients response (not closing connection after receiving 302) * RST seen in 19404 packet is the Load Balancer forcing to reset this CP stream due to the timeout
(08 Aug '12, 00:33) andrus
Look at who is closing the connection. The first FIN comes from x.251.21.95. It is the loadbalancer that is closing the connection not the client (first screenshot). So, the reason for the different behaviour is clearly the firmware upgrade on the loadbalancer. Maybe there is a new feature in the new firmare that keeps the connection open to the client. Unfortunately the new client request (to the redirected URL) is not part of the capture screenshot. Do you have any performance issues with this behaviour, or do you just want to know why it happens (firmware upgrade)? (08 Aug '12, 01:20) Kurt Knochner ♦ Spot on Kurt, I have no idea how could I missed that ! I will move away from the client toward the load balancer checking for any differences in TCP/HTTP profiles in the new version. Yes it is performance issue. Basically after upgrading LB the login process takes 2-3 min longer when going thought upgraded LB. Here is trace taken on the LB directly (filtered for the front-end communication for this client) - 192.168.1.172 is the LB VIP ip address - clients POST was toggled as a time 0
I would have though that if the client received 302 with "Connection:Close" header - it should terminate the existing TCP stream and initiate new TCP connection for the URL given in the "Location:" header - it's that the case ?? HTTP/1.1 302 Found Server: My Own Date: Fri, 03 Aug 2012 10:00:02 GMT Connection: close Location: http://www.xxxxxx.xx/mail/testuser.nsf Tanks for you help Andrzej (08 Aug '12, 02:22) andrus
Never mind. You know, sometimes ...
yes, please check that.
Look at both POST requests with "Follow TCP stream" and compare anything in both connections. Is there any difference at HTTP protocol level? If there is none, the loadbalancer behaves just different. If there is a difference, check if the LB answer (new firmware) is HTTP compliant. If it is not, the client might just wait for more data within the open connection, until it gets closed by the LB. The 186 seconds to the connection close, sounds like your delay of 2-3 minutes. (08 Aug '12, 02:35) Kurt Knochner ♦
well, yes and no ;-) Take a look at the HTTP 1.1 RFC. It states:
The server can close the connection when it finished sending all data. The client however, can only close the connection when it received all data. If the "Content-Length" header contains the wrong length, the client might just wait for the rest of the data until the LB closes the connection. As the problem only occurs after the LB upgrade, there must be some change in behavior or a bug with the LB. (08 Aug '12, 02:47) Kurt Knochner ♦ |
My guess would be that the 302 response is not correct. Is there a Content-Length header? And does it correctly state the length of the 302 response? Is Keep-Alive used by any chance? It's pretty hard to analyze this issue based on just a screenshot, that's why we all love to use Wireshark :-) As you obfuscated some of the info, I assume you are not able to share the tracefiles? Are you able to send them privately to one or two persons? I'd be willing to have a quick look at them if you'd like. answered 08 Aug '12, 02:21  SYN-bit ♦♦ If: Are all the headers that the client receives, it will indeed wait. As there is no "Content-Length" header. This means the client does not know when the object has been sent completely, so it needs to wait for the FIN from the server (which it normally sends when "Connection: close" is being used. You might want to take a look at the server side to see whether the server indeed closes the connection and then check the settings on the LB why it is not closing the connection. If it wants to keep the connection open, it should add a "Content-Length: 0" header to the client. (08 Aug '12, 02:33) SYN-bit ♦♦ Thanks a mil for your input on this. 302 looks like this (there is no Content-Lenght header) This is the 302 given to the client:HTTP/1.1 302 Found Server: My Own Date: Fri, 03 Aug 2012 09:51:13 GMT Connection: close Location: http://www.xxxxxx.xx/mail/testuser.nsf Set-Cookie: LtpaToken=AHDER7LK9kHknb8MJHGRT876KNJGTFD0KHJSUJNBGTc2aL6c8v0XQ35wDW14B8z85Gv; domain=.xxxxxx.xx; path=/ (08 Aug '12, 02:44) andrus I have now checked the in the traces taken on the back-end server and can confirm in both scenarios that the back-end after responding with 302 to the initial POST is closing the existing connection
I will check on the LB why there is hold up for so long in terminating this connection after 302 is passed to the client... Thank you all for looking into this. Andrzej (08 Aug '12, 03:15) andrus |
Did you ever find an answer on this? I’m having pretty much the same exact issue and I’m using Citrix Netscalers. I’m having 3-5 minute delays when moving between tabs in one of our lotus apps. Let me know please!
Yes, I did - it was eventually looked at by the engineering team of the LB vendor and there was some issue with the web application firewall found, later fixed in the next firmware release :) so make sure first you are using the latest build just in case :)
Thanks for the reply. I eventually found my problem as well. With the Citrix Netscaler’s in the code I was running they forgot to include a piece of code that sent the Close Notify for SSL sessions. Without this close notify my session stayed open until the timeout and then would reset and work. They verified this was missing and is in the latest code of the Netscaler.