This is our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

Hi, We're trying to trouble shoot a recent change in our load test results. A full second of time has been added to each request. Using Wireshark to look at a packet capture from the web server, we saw the delay coming from the server/requester side.

What appears to be happening if the SYN, SYN/ACK, ACK conversation finishes in basically 0 seconds. The load server's GET request waits almost a full second before it's sent.

It seems to be a recent change in behavior on this Windows 2003 virtual server.

Have you run into any similar, or have an idea what may be happening? The only thing I've been able to find at this point is the "TCPNoDelay" registry setting in Windows.

Thanks! Ben

asked 03 Jul '12, 07:02

doubtingben's gravatar image

doubtingben
1112
accept rate: 0%


Hard to say without having a trace file to look at. Maybe you could upload a sample to Cloudshark?

And by the way... in network analysis a second is an eternity, so saying that the three way handshake takes basically 0 seconds isn't exactly exact ;-) For a local LAN setup it should be below 1 Millisecond.

And I doubt a registry setting will help much... every stack tries to answer packets as quick as possible, so if your response times have gone up drastically it is usually an overload situation on the server (CPU/Memory/Disk constraints)

permanent link

answered 03 Jul '12, 07:10

Jasper's gravatar image

Jasper ♦♦
23.8k551284
accept rate: 18%

edited 03 Jul '12, 07:13

We figured overload was the issue also and moved all the app's servers to the same subnet. The load generator and target web server are both on the same vm host now.

Cloudshark is awesome! First time I heard about it.

Here's a stream: http://www.cloudshark.org/captures/ff416b3b6ec5

(03 Jul '12, 07:17) doubtingben

Thank you for the trace. Funny thing is that the delta times between packets are mostly all zero, so I guess they're below the microsecond range and not distinguishable - which would mean that things work quite fast in most cases.

Packet 4 is indeed interesting - almost 1 second for the client to start asking the server for content after the handshake was established. That is an eternity, and I guess it's because the client has to generate the request first (which is actually one of the longest requests I have seen in a while... it is spanning two packets).

The Server starts replying after 15ms which isn't great, but I guess it has to pull ressources from a database itself, so the time isn't critical.

Conclusion: it looks like the client takes a long time to get its GET requests prepared. It is not the server who is slowing things down. You need to find out why the client takes this long to send the GET requests. It is also not a network issue, since transport works incredibly fast and without any serious packet loss.

(03 Jul '12, 07:32) Jasper ♦♦

Thanks for looking at it. That's basically what I came up with, but wasn't that confident :)

We'll look at the load generator server some more, maybe try a different tool.

(03 Jul '12, 07:38) doubtingben

You say, you've seen a difference in load test results.

A full second of time has been added to each request.

Where did you measure that time? Did you run the load test software on the same virtual machine?

Reason why I'm asking: I don't trust the timestamp values in the capture file.

Take a look a this: frame.number >=8 and frame.number < 105

During that time (the SAME timestamp of 0.984375 in that period) there were 96502 bytes transfered (according to Wireshark Statistics -> Summary). Even if the last frame were at 0.984376 (it is NOT), that would mean 96502 bytes in 0.000001 seconds, which would be 96502000000 bytes /second (96502 MByte/s!!). I doubt that this is possible, even in VMware ;-).

So there is something wrong with the timestamps OR with the way the OS within VMware provides those timestamps. If you measured the time for the load test on the same system, the "missing second" could be just a result of the same bad timestamps.

Just an idea ...

Regards
Kurt

permanent link

answered 03 Jul '12, 09:54

Kurt%20Knochner's gravatar image

Kurt Knochner ♦
24.8k1039237
accept rate: 15%

edited 03 Jul '12, 10:55

Where did you measure that time? This is coming from the commercial load generating software.

Did you run the load test software on the same virtual machine? Yes, we started seeing erratic results across a firewall, so we moved the vm to a new host behind the firewall. The erratic spikes disappeared, but this 1s persists.

I see what you're saying about the timestamps. I didn't really look at anything after the GET because from GET to the HTTP 200, the total time looks the same as expected.

(03 Jul '12, 10:24) doubtingben

Yes, we started seeing erratic results across a firewall, so we moved the vm to a new host behind the firewall. The erratic spikes disappeared, but this 1s persists.

As I said, I don't trust the timestamps. And if there is a problem with wireshark, there might be a problem with the results of the load test as well. Can you test from a different machine on the LAN?

(03 Jul '12, 10:47) Kurt Knochner ♦
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×752
×254
×43
×9

question asked: 03 Jul '12, 07:02

question was seen: 3,827 times

last updated: 03 Jul '12, 10:55

p​o​w​e​r​e​d by O​S​Q​A