Counting incoming TCP open & http GET requests to a web server

Question

I want to use wireshark (preferably tshark) as a sniffer for web server performance symbiotic simulation analysis. The countables of interest are:

Incoming TCP open connections i.e. "tcp.flags.syn==1" and "tcp.flags.ack==0"
Incoming GET http requests.

For those I don't need the details, only counters (quantities). These need to be stored in the same file, with some text readable format, because another application will be reading and producing output from them.

Does anyone know of a way to do that?

Answer 1

4

Here is a way to count the tcp flags and HTTP get requests:
TShark statistics.

$ tshark -r Clmt_04.pcap -qz "io,stat,0,COUNT(tcp.flags)tcp.flags==0x12" -z "io,stat,0,COUNT(http.request.method)http.request.method=="GET""
===================================================================
IO Statistics
Column #0: COUNT(http.request.method)http.request.method==GET
                |   Column #0
Time            |          COUNT
000.000-                      115
===================================================================
===================================================================
IO Statistics
Column #0: COUNT(tcp.flags)tcp.flags==0x12
|   Column #0
Time            |          COUNT
000.000-                       74
===================================================================

answered 18 Dec ‘11, 08:26

joke
1.3k●4●9●34
accept rate: 9%

a.
This was very helpful, thank you.
I use a wireshark capture file that was filtered ‘tcp dst port 8080’.
The above code runs to completion, but although the http GET count turns up correct, the tcp.flags count turns up 0.
Is the value tcp.flags==0x12 correct for capturing tcp open requests?
I’d thought that only the tcp-syn flag would be set in such packets, although I can’t tell which value that corresponds to in the tcp.flags variable.

b.
Ideally I’d like to have a process that counts such packets in real time and produces a simple (capture?) file where it updates the two packet counts of interest in intervals.
Is there any way to configure tshark to do that?

(19 Dec ‘11, 10:52) adonies

a
Sorry, my bad, should be<br< tcp.flags==0x02

(19 Dec ‘11, 12:59) joke

With this value, I get a count of tcp.flags(0x02) 41.478 in my capture file, while the http.request.method(GET) is at 33.280.
This result is strange because these counts should have been more or less the same.
The web client requesting the data from the web server is custom-made and implements http/0.1 only: open TCP connection, request file, get data, close TCP.
I know from the web server logs that about 33.280 were serviced 200 OK, and from the web client logs that about 33.528 requests were sent.
I expected a count of about 33.528 tcp.flags(0x02) instead of 41.478.

(20 Dec ‘11, 14:20) adonies

So I’m assuming the traffic you’re capturing has only the test sequences, so there aren’t other TCP connections being requested, possibly for protocols other than HTTP or for HTTP where the first request isn’t a GET.

If so, what happens if you look instead for SYN+ACK from the server? If the initial SYN doesn’t get responded to by the server, you’ll get an initial SYN but you won’t get a TCP connection and thus won’t get a GET request.

(22 Dec ‘11, 15:17) Guy Harris ♦♦

Answer 2

io,stat
You can send the output only to a text file.

$ tshark -i 3 -qz "io,stat,120,COUNT(tcp.flags)tcp.flags==0x02" -z "io,stat,120,COUNT(http.request.method)http.request.method=="GET"" > io-stat.txt
===================================================================
io-stat.txt
===================================================================
IO Statistics
Interval: 120.000 secs
Column #0: COUNT(http.request.method)http.request.method==GET
                |   Column #0
Time            |          COUNT
000.000-120.000                 0
120.000-240.000               151
===================================================================
===================================================================
IO Statistics
Interval: 120.000 secs
Column #0: COUNT(tcp.flags)tcp.flags==0x02
                |   Column #0
Time            |          COUNT
000.000-120.000                 0
120.000-240.000                83
===================================================================

Output to capture file
You cannot use display filters, when capturing and saving the captured packets. So you have to use capture filters.

Capture filters:
Capture SYN packets:
tcp[0xd]&18=2

Capture GET requests:
tcp[20:4]=0x47455420

$ tshark -i 3 -f "tcp[13]=0x02 or tcp[20:4]=0x47455420" -w syn-http.request.get.pcap

Useful links:
http://www.tcpdump.org/tcpdump_man.html
http://www.packetlevel.ch/html/tcpdumpf.html

Answer 3

If your webserver tells you it served 33.280 objects and wireshark tells you it saw 33.280 http requests then all requests that were on the wire did get a response. Now your client tells you it requested 33.528 objects. So in 248 cases it was not able to get the request on the wire.

Combine that with the fact that you saw 41.478 TCP/SYN packets, then you can imagine that there was a problem opening TCP sessions to the server. This can be either because the server was too busy to handle all the incoming connections, but this seems illogical as all HTTP requests were properly answered. It could also be that the webserver has a configured limit on the amount of concurrent sessions it can handle and you hit that limit. This would explain the SYN retransmissions and when it fails to get a TCP connection at all, the missing requests.

BTW It's better to use the capture filter "tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420" to capture the GET requests, as this filter will work even when there are TCP options in the packet (it looks at the TCP header length and skips the proper amount of bytes to get to the TCP payload).

Answer 4

Thank you joke, helloworld, SYNbit and Guy Harris for your answers. I think I got the problem under control now.