Counting incoming TCP open & http GET requests to a web server

Question

I want to use wireshark (preferably tshark) as a sniffer for web server performance symbiotic simulation analysis. The countables of interest are:

Incoming TCP open connections i.e. "tcp.flags.syn==1" and "tcp.flags.ack==0"
Incoming GET http requests.

For those I don't need the details, only counters (quantities). These need to be stored in the same file, with some text readable format, because another application will be reading and producing output from them.

Does anyone know of a way to do that?

Answer 1

4

Here is a way to count the tcp flags and HTTP get requests:
TShark statistics.

$ tshark -r Clmt_04.pcap -qz "io,stat,0,COUNT(tcp.flags)tcp.flags==0x12" -z "io,stat,0,COUNT(http.request.method)http.request.method=="GET""
===================================================================
IO Statistics
Column #0: COUNT(http.request.method)http.request.method==GET
                |   Column #0
Time            |          COUNT
000.000-                      115
===================================================================

===================================================================
IO Statistics
Column #0: COUNT(tcp.flags)tcp.flags==0x12
                |   Column #0
Time            |          COUNT
000.000-                       74
===================================================================

permanent link

answered 18 Dec '11, 08:26

joke
1.3k●4●9●34
accept rate: 9%

a.
This was very helpful, thank you.
I use a wireshark capture file that was filtered 'tcp dst port 8080'.
The above code runs to completion, but although the http GET count turns up correct, the tcp.flags count turns up 0.
Is the value tcp.flags==0x12 correct for capturing tcp open requests?
I'd thought that only the tcp-syn flag would be set in such packets, although I can't tell which value that corresponds to in the tcp.flags variable.

b.
Ideally I'd like to have a process that counts such packets in real time and produces a simple (capture?) file where it updates the two packet counts of interest in intervals.
Is there any way to configure tshark to do that?

(19 Dec '11, 10:52) adonies

a
Sorry, my bad, should be<br< tcp.flags==0x02

(19 Dec '11, 12:59) joke

With this value, I get a count of tcp.flags(0x02) 41.478 in my capture file, while the http.request.method(GET) is at 33.280.
This result is strange because these counts should have been more or less the same.
The web client requesting the data from the web server is custom-made and implements http/0.1 only: open TCP connection, request file, get data, close TCP.
I know from the web server logs that about 33.280 were serviced 200 OK, and from the web client logs that about 33.528 requests were sent.
I expected a count of about 33.528 tcp.flags(0x02) instead of 41.478.

(20 Dec '11, 14:20) adonies

So I'm assuming the traffic you're capturing has only the test sequences, so there aren't other TCP connections being requested, possibly for protocols other than HTTP or for HTTP where the first request isn't a GET.

If so, what happens if you look instead for SYN+ACK from the server? If the initial SYN doesn't get responded to by the server, you'll get an initial SYN but you won't get a TCP connection and thus won't get a GET request.

(22 Dec '11, 15:17) Guy Harris ♦♦

Answer 2

2

io,stat
You can send the output only to a text file.

$ tshark -i 3 -qz "io,stat,120,COUNT(tcp.flags)tcp.flags==0x02" -z "io,stat,120,COUNT(http.request.method)http.request.method=="GET"" > io-stat.txt
===================================================================
io-stat.txt
===================================================================
IO Statistics
Interval: 120.000 secs
Column #0: COUNT(http.request.method)http.request.method==GET
                |   Column #0
Time            |          COUNT
000.000-120.000                 0
120.000-240.000               151
===================================================================
===================================================================
IO Statistics
Interval: 120.000 secs
Column #0: COUNT(tcp.flags)tcp.flags==0x02
                |   Column #0
Time            |          COUNT
000.000-120.000                 0
120.000-240.000                83
===================================================================

Output to capture file
You cannot use display filters, when capturing and saving the captured packets. So you have to use capture filters.

Capture filters:
Capture SYN packets:
tcp[0xd]&18=2

Capture GET requests:
tcp[20:4]=0x47455420

$ tshark -i 3 -f "tcp[13]=0x02 or tcp[20:4]=0x47455420" -w syn-http.request.get.pcap

Useful links:
http://www.tcpdump.org/tcpdump_man.html
http://www.packetlevel.ch/html/tcpdumpf.html

permanent link

answered 19 Dec '11, 12:56

joke
1.3k●4●9●34
accept rate: 9%

edited 19 Dec '11, 12:58

io,stat
a. The command line process does not end on its own, but keeps recording in 120 sec intervals, until killed (Ctrl-C) - and then only writes to the output file. Is there an option to have the process end on its own?
b. Can I add the option -f "tcp dst port 8080" in the command line as well?

output to capture file
a. Can I add the "tcp dst port 8080" in capture filter?
b. The "syn-http.request.get.pcap" file is written continually or at the end?
c. Can I use the -r option with that file to count tcp & http, as described in your previous post?

(20 Dec '11, 14:36) adonies

1

The command line process does not end on its own, but keeps recording in 120 sec intervals, until killed (Ctrl-C) - and then only writes to the output file. Is there an option to have the process end on its own?

To stop the capture after a timeout, use tshark's -a flag with duration:

$ tshark -a duration:120

(20 Dec '11, 15:15) helloworld

io,stat
a. Thanks helloworld:)
For Statistics you can also use 0 (=no interval)
tshark -i 3 -qz "io,stat,0,COUNT(tcp.flags)tcp.flags==0x02"
b. Yes: tshark -i 3 -f "tcp dst port 8080" -qz "io,stat,120,COUNT(tcp.flags)tcp.flags==0x02" -z "io,stat,120,COUNT(http.request.method)http.request.method=="GET"" > io-stat-dstport8080.txt

(20 Dec '11, 22:19) joke

output to capture file
a. tshark -i 3 -f "(tcp dst port 8080) && (tcp[13]=0x02 or tcp[20:4]=0x47455420)" -w syn-http.request.get.dstport8080.pcap
b.
c. You can run TShark twice (or even more):
use one instance to capture the traffic and another instance for statistics

(20 Dec '11, 22:21) joke

Answer 3

If your webserver tells you it served 33.280 objects and wireshark tells you it saw 33.280 http requests then all requests that were on the wire did get a response. Now your client tells you it requested 33.528 objects. So in 248 cases it was not able to get the request on the wire.

Combine that with the fact that you saw 41.478 TCP/SYN packets, then you can imagine that there was a problem opening TCP sessions to the server. This can be either because the server was too busy to handle all the incoming connections, but this seems illogical as all HTTP requests were properly answered. It could also be that the webserver has a configured limit on the amount of concurrent sessions it can handle and you hit that limit. This would explain the SYN retransmissions and when it fails to get a TCP connection at all, the missing requests.

BTW It's better to use the capture filter "tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420" to capture the GET requests, as this filter will work even when there are TCP options in the packet (it looks at the TCP header length and skips the proper amount of bytes to get to the TCP payload).

Answer 4

0	Thank you joke, helloworld, SYNbit and Guy Harris for your answers. I think I got the problem under control now. permanent link answered 22 Dec '11, 12:43 adonies 12●2●2●5 accept rate: 0% edited 27 Dec '11, 10:53

Counting incoming TCP open & http GET requests to a web server

Follow this question

You have a trillion packets.

You need to see four of them.

Don't have Wireshark?

Related questions