Display “Uncompressed entity body” like old tshark did with the -x option

Question

In tshark 1.6.7 the command

tshark -i lo -x -R 'http.response.code == 302' -l

resulted in an output like the following:

26.047812 127.0.0.1 -> 127.0.0.1 HTTP 798 HTTP/1.1 302 Found (text/html) Frame (798 bytes): 0000 00 00 00 00 00 00 00 00 00 00 00 00 08 00 45 00 …………..E. 0010 03 10 1a 6a 40 00 40 06 1f 7c 7f 00 00 01 7f 00 [email protected]@..|…… 0020 00 01 00 50 c4 51 20 88 1b 52 89 29 63 e3 80 18 …P.Q ..R.)c… 0030 02 00 01 05 00 00 01 01 08 0a 00 1d 01 41 00 1d ………….A.. 0040 01 3b 48 54 54 50 2f 31 2e 31 20 33 30 32 20 46 .;HTTP/1.1 302 F 0050 6f 75 6e 64 0d 0a 44 61 74 65 3a 20 4d 6f 6e 2c ound..Date: Mon, [many more lines like this …] Uncompressed entity body (1069 bytes):

0000 3c 70 72 65 20 63 6c 61 73 73 3d 27 78 64 65 62 fpre class='xdeb 0010 75 67 2d 76 61 72 2d 64 75 6d 70 27 20 64 69 72 ug-var-dump' dir 0020 3d 27 6c 74 72 27 3e 0a 3c 62 3e 61 72 72 61 79 ='ltr'>.<b>array 0030 3c 2f 62 3e 0a 20 20 27 61 72 74 4e 61 6d 65 27 </b>. 'varnamee' [many more lines like this …]

Is it possible to get that output with tshark 1.12.3 as well? Especially the part that begins with “Uncompressed entity body” is important for me because I pipe tshark’s output to a script which converts that part into a readable text.

I read the release notes at https://www.wireshark.org/docs/relnotes/wireshark-1.10.0.html. They suggest using -Px instead of x. So my current command looks like

tshark -i lo -Px -Y 'http.response.code == 302' -l

But that command does not show the uncompressed entity body.

Accepted Answer

2

The command line tshark -r input.pcapng -x -l -Y 'http.response.code == 200' works for me with tshark 1.12.3, I get the output below:

Frame (792 bytes):
0000  74 d0 2b 9e 2d 54 c0 c1 c0 78 70 c7 08 00 45 00   t.+.-T...xp...E.
...
0310  00 0d 0a 30 0d 0a 0d 0a                           ...0....
Reassembled TCP (5094 bytes):
0000  48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d   HTTP/1.1 200 OK.
...
13e0  0a 30 0d 0a 0d 0a                                 .0....
De-chunked entity body (4686 bytes):
0000  1f 8b 08 00 00 00 00 00 00 03 cd 5b 6d 73 1b 37   ...........[ms.7
...
1240  fb 4e cf f5 ff 00 7d 6e 14 c7 ec 44 00 00         .N....}n...D..
Uncompressed entity body (17644 bytes):
0000  76 61 72 20 63 75 72 5f 74 6f 70 69 63 5f 69 64   var cur_topic_id
...

Can you post a small capture that exhibits the problem somewhere public, e.g. Cloudshark, Google Drive, Dropbox?

answered 24 Jan '15, 04:39

grahamb ♦
19.8k●3●30●206
accept rate: 22%

@grahamb You are refering to a situation in which a previously written capture file is read. In that case my output is similar to yours. What I am looking for is a solution with live capturing. The command that I mentioned could be entered on the command line and each time a package with http response code 302 is found it is written into the the terminal immediately. Is that still possible? Thanx for your answer.

(24 Jan '15, 06:23) miachino

Interesting, I tried it on a live capture redirecting the output to a file, tshark -i 4 -x -l -Y 'http.response.code == 200' > c:\temp\xx.txt and got the expected output.

If I just print to the console though, I don't see the "headers" at the start of each part of the output. Very odd, nor do I see the entity body in the hex.

(25 Jan '15, 09:24) grahamb ♦

@grahamb Thanx for testing. In my case redirecting tshark's output to a file or printing the output directly to the terminal doesn't make a difference. I do not get the part "Uncompressed entity body" in both cases. Does that mean that it is not possible anymore to get the uncompressed entity body with tshark in a live capture? (I use arch linux.)

(26 Jan '15, 08:04) miachino

It must be possible as I managed it (on Windows) when redirecting to a file. In your intended use you are redirecting the output somewhere though?

(26 Jan '15, 08:21) grahamb ♦

just a thought, in the default Wireshark profile you do have all the http dissector reassemble options enabled, along with "Uncompress entity bodies" and the tcp dissector option "Allow subdissector to reassemble TCP streams".

(26 Jan '15, 08:35) grahamb ♦

You can check the prefs used in your tshark invocation by using:

tshark -G currentprefs | <filter for "http.">

replacing the filter method string with whatever is appropriate for your environment. I get:

#http.desegment_headers: TRUE
#http.desegment_body: TRUE
#http.dechunk_body: TRUE
#http.decompress_body: TRUE
#http.tcp.port: 80,3128,3132,5985,8080,8088,11371,1900,2869,2710
#http.ssl.port: 443

The other one to check is "tcp.desgment_tcp_streams", I have:

#tcp.desegment_tcp_streams: TRUE

(26 Jan '15, 08:50) grahamb ♦

There's a site httpbin.org that allows you to test certain http responses. Using the request http://httpbin.org/gzip and a tshark capture line of:

ts -i 2 -x -l -f "host httpbin.org" -Y 'http.response.code == 200'

I see the uncompressed data:

Capturing on 'Ethernet' Frame (670 bytes): 0000 f8 b1 56 e1 c0 8b 00 d0 cf 03 7c 30 08 00 45 80 ..V.......|0..E. 0010 02 90 f7 93 40 00 2f 06 ab 00 36 af de f6 0a 9e [email protected]/...6..... 0020 86 10 00 50 d0 58 f5 27 83 4c 28 be 87 ad 50 18 ...P.X.'.L(...P. 0030 01 12 f1 5f 00 00 48 54 54 50 2f 31 2e 31 20 32 ..._..HTTP/1.1 2 0040 30 30 20 4f 4b 0d 0a 53 65 72 76 65 72 3a 20 6e 00 OK..Server: n 0050 67 69 6e 78 0d 0a 44 61 74 65 3a 20 4d 6f 6e 2c ginx..Date: Mon, 0060 20 32 36 20 4a 61 6e 20 32 30 31 35 20 31 37 3a 26 Jan 2015 17: 0070 31 31 3a 30 38 20 47 4d 54 0d 0a 43 6f 6e 74 65 11:08 GMT..Conte ... 0280 f3 88 66 49 c5 63 2e 4a ac 6f b4 2e f6 1c 4c bb ..fI.c.J.o....L. 0290 be d5 b8 37 7e 00 54 a8 81 25 1b 02 00 00 ...7~.T..%.... Uncompressed entity body (539 bytes): 0000 7b 0a 20 20 22 67 7a 69 70 70 65 64 22 3a 20 74 {. "gzipped": t 0010 72 75 65 2c 20 0a 20 20 22 68 65 61 64 65 72 73 rue, . "headers 0020 22 3a 20 7b 0a 20 20 20 20 22 41 63 63 65 70 74 ": {. "Accept ...

(26 Jan '15, 09:15) grahamb ♦

@grahamb Yes, I used tshark 1.6.7 in a command like the following:

tshark -i lo -x -R 'http.response.code == 302' -l | scriptThatParsesTsharksOutput.php

That does not work anymore because my php script expects the input that was delivered by tshark 1.6.7.

To your other question: I didn't have wireshark installed, only tshark. Do settings in wireshark effect how tshark works?

Because of your question I installed wireshark but I do not have a default profile I think. In wireshark I click Edit -> Configuration Profiles. In the popup window I see three profiles: Default, Bluetooth and Classic. For each profile a directory name is given. For the profile "Default" that directory name is ~/.wireshark/profiles but that directory does not exist. I only have the directory ~/.wireshark with the two files recent and recent_common. That's why I said that I think that I do not have a default profile.

(27 Jan '15, 03:03) miachino

@grahamb Ah, I did not see your last two comments before I wrote my comment because they where hidden and only visible after clicking "show 2 more comments". I will check that.

(27 Jan '15, 03:09) miachino

I just repeated my test to http://httpbin.org/gzip, piping tshark output to another program and got the expected text.

Can you try that?

(27 Jan '15, 03:38) grahamb ♦

@grahamb Testing with that website was a very good idea because that test showed me that I indeed get the expected result as well. The reason why I did not get the part "Uncompressed entity body" was because I performed my tests with a page, that was very short and in that case the Apache web server does not compress the response as I found out at http://httpd.apache.org/docs/trunk/upgrading.html. There I found the note: "mod_deflate will now skip compression if it knows that the size overhead added by the compression is larger than the data to be compressed." That was the case in my situation. So there simply was no compressed part. Thank you very much for your kind help and for your patience.

Now I have another related question but I guess that it's better to ask that in a separate question.

(27 Jan '15, 17:34) miachino

showing 5 of 11 show 6 more comments