Hi, We're trying to trouble shoot a recent change in our load test results. A full second of time has been added to each request. Using Wireshark to look at a packet capture from the web server, we saw the delay coming from the server/requester side. What appears to be happening if the SYN, SYN/ACK, ACK conversation finishes in basically 0 seconds. The load server's GET request waits almost a full second before it's sent. It seems to be a recent change in behavior on this Windows 2003 virtual server. Have you run into any similar, or have an idea what may be happening? The only thing I've been able to find at this point is the "TCPNoDelay" registry setting in Windows. Thanks! Ben asked 03 Jul '12, 07:02  doubtingben |
2 Answers:
Hard to say without having a trace file to look at. Maybe you could upload a sample to Cloudshark? And by the way... in network analysis a second is an eternity, so saying that the three way handshake takes basically 0 seconds isn't exactly exact ;-) For a local LAN setup it should be below 1 Millisecond. And I doubt a registry setting will help much... every stack tries to answer packets as quick as possible, so if your response times have gone up drastically it is usually an overload situation on the server (CPU/Memory/Disk constraints) answered 03 Jul '12, 07:10  Jasper ♦♦ edited 03 Jul '12, 07:13 |
You say, you've seen a difference in load test results.
Where did you measure that time? Did you run the load test software on the same virtual machine? Reason why I'm asking: I don't trust the timestamp values in the capture file. Take a look a this: During that time (the SAME timestamp of 0.984375 in that period) there were 96502 bytes transfered (according to Wireshark Statistics -> Summary). Even if the last frame were at 0.984376 (it is NOT), that would mean 96502 bytes in 0.000001 seconds, which would be 96502000000 bytes /second (96502 MByte/s!!). I doubt that this is possible, even in VMware ;-). So there is something wrong with the timestamps OR with the way the OS within VMware provides those timestamps. If you measured the time for the load test on the same system, the "missing second" could be just a result of the same bad timestamps. Just an idea ... Regards answered 03 Jul '12, 09:54  Kurt Knochner ♦ edited 03 Jul '12, 10:55
I see what you're saying about the timestamps. I didn't really look at anything after the GET because from GET to the HTTP 200, the total time looks the same as expected. (03 Jul '12, 10:24) doubtingben
As I said, I don't trust the timestamps. And if there is a problem with wireshark, there might be a problem with the results of the load test as well. Can you test from a different machine on the LAN? (03 Jul '12, 10:47) Kurt Knochner ♦ |
We figured overload was the issue also and moved all the app's servers to the same subnet. The load generator and target web server are both on the same vm host now.
Cloudshark is awesome! First time I heard about it.
Here's a stream: http://www.cloudshark.org/captures/ff416b3b6ec5
Thank you for the trace. Funny thing is that the delta times between packets are mostly all zero, so I guess they're below the microsecond range and not distinguishable - which would mean that things work quite fast in most cases.
Packet 4 is indeed interesting - almost 1 second for the client to start asking the server for content after the handshake was established. That is an eternity, and I guess it's because the client has to generate the request first (which is actually one of the longest requests I have seen in a while... it is spanning two packets).
The Server starts replying after 15ms which isn't great, but I guess it has to pull ressources from a database itself, so the time isn't critical.
Conclusion: it looks like the client takes a long time to get its GET requests prepared. It is not the server who is slowing things down. You need to find out why the client takes this long to send the GET requests. It is also not a network issue, since transport works incredibly fast and without any serious packet loss.
Thanks for looking at it. That's basically what I came up with, but wasn't that confident :)
We'll look at the load generator server some more, maybe try a different tool.