This is a static archive of our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

tshark -T fields question

0

Hi, We have some web services that we'd like to capture packets for using tshark. Some of the service traffic is dissected using -R xml argument and some are dissected using -R data argument. We like to see the payload using the -T fields argument. So with -R data we run the following command. tshark -i 1 -R data -T fields -e frame - frame.time -e ip.src -e ip.dst -e tcp -e data -V -l the output for this is something like this

Frame 538: 814 bytes on wire (6512 bits), 814 bytes captured (6512 bits)    May 22, 2012 11:52:49.933924000 192.168.0.19    192.168.0.16    Transmission Control Protocol, Src Port: sunproxyadmin (8081), Dst Port: 35946 (35946), Seq: 1, Ack: 745, Len: 748  485454502f312e312035303020496e7465726e616c20536572

We'd like to run a similar command to get the actual text of the XML So to get an ouput such as

Frame 538: 814 bytes on wire (6512 bits), 814 bytes captured (6512 bits)    May 22, 2012 11:52:49.933924000 192.168.0.19    192.168.0.16    Transmission Control Protocol, Src Port: sunproxyadmin (8081), Dst Port: 35946 (35946), Seq: 1, Ack: 745, Len: <test>mytest</test>

We tried something like this tshark -i 1 -R xml -T fields -e frame - frame.time -e ip.src -e ip.dst -e tcp -e xml -V -l but we didn't get the desired output. IS there a way to run -T fields command to get the xml payload in the output?

asked 25 May '12, 07:05

aaghili's gravatar image

aaghili
16336
accept rate: 0%

edited 25 May '12, 08:37

grahamb's gravatar image

grahamb ♦
19.8k330206


3 Answers:

0

Can you try this:

tshark -i 1 -R xml -T fields -e <your other fields> -e text

Please check this question (similar problem).

Regards
Kurt

answered 25 May '12, 07:08

Kurt%20Knochner's gravatar image

Kurt Knochner ♦
24.8k1039237
accept rate: 15%

edited 25 May '12, 07:24

Thank you for the response. When I set it up as you said I don't see any xml text. Its just blank

(25 May '12, 10:37) aaghili

this is what I get. HAve you tested this and it works for you? Frame 117 (851 bytes on wire, 851 bytes captured) May 25, 2012 10:35:11.582303000 10.203.192.79 10.202.160.99 Transmission Control Protocol, Src Port: cdid (3315), Dst Port: http (80), Seq: 55761, Ack: 1, Len: 797

(25 May '12, 10:38) aaghili

0

This is what I get, when I load this page:

URL:

http://www.w3schools.com/xml/simple.xml

Command:

tshark.exe" -i 3 -T fields -e frame.number -e frame.time -e ip.src -e ip.dst -e data -e text port 80

Output:

Capturing on LAN 1 May 25, 2012 19:46:55.097596000 192.168.90.51 66.29.212.73 GET /xml/simple.xml HTTP/1.1\r\n,\r\n 2 May 25, 2012 19:46:55.243759000 66.29.212.73 192.168.90.51 HTTP/1.1 200 OK\r\n,Accept-Rang es: bytes\r\n,ETag: "e5309bb5a12ecd1:0"\r\n,Vary: Accept-Encoding\r\n,X-Powered- By: ASP.NET\r\n,\r\n,Content-encoded entity body (gzip): 574 bytes -> 1135 bytes ,?>,</name>,</price>,</description>,</calories>,</food>,</name>,</price>,</descr iption>,</calories>,</food>,</name>,</price>,</description>,</calories>,</food>, </name>,</price>,</description>,</calories>,</food>,</name>,</price>,</descripti on>,</calories>,</food>,</breakfast_menu> 3 May 25, 2012 19:46:55.443582000 192.168.90.51 66.29.212.73

tshark Version:
TShark 1.7.1 (SVN Rev 41970 from /trunk) (Windows 7 64 Bit).

Questions:

  1. What's your version of tshark?
  2. Can you please try the command above (without -R xml)?
  3. Can you post a sample capture on cloudshark.org?

Regards
Kurt

answered 25 May '12, 10:53

Kurt%20Knochner's gravatar image

Kurt Knochner ♦
24.8k1039237
accept rate: 15%

edited 25 May '12, 11:08

I get the same result. its blank. This service does get dissected correctly as XML when I run this command. tshark -i 7 -R xml -V -l When I run this command I can see the packet. The issue is the XML is so large that this outputs over 20000 lines. The reason I'd like to use the -T fields is so I can get one single line that's very large instead of 20000 small lines. Our app reads the standard out of tshark output. Thank you for the help!

(25 May '12, 11:09) aaghili

the version of tshark is 1.2.10 running on linux 64bit.

(25 May '12, 11:11) aaghili

can you upgrade tshark?

(25 May '12, 11:20) Kurt Knochner ♦

sure where do you download the latest linux version 64bit?

(25 May '12, 14:40) aaghili

Use your distributions package manager, or compile your own. Compilation details can be found here.

(25 May '12, 14:46) grahamb ♦

ok I now get what you're getting but that is not the valid xml that is being sent back. take a look at your xml text. That's just some tags not the actual response for the site. Is there a way to see the actual payload of the xml response?

(26 May '12, 06:22) aaghili

That's the response of the server. It contains the HTTP response headers and the data (XML). If you remove the HTTP headers (with a script) you will get the XML data.

(26 May '12, 09:01) Kurt Knochner ♦

ok I now get what you're getting but that is not the valid xml that is being sent back. take a look at your xml text. That's just some tags not the actual response for the site. Is there a way to see the actual payload of the xml response?

Here what the payload is <?xml version="1.0" encoding="ISO-8859-1"?><!-- Edited by XMLSpy® --><breakfast_menu> <food><name>Belgian Waffles</name><price>$5.95</price><description>two of our famous Belgian Waffles with plenty of real maple syrup</description> <calories>650</calories></food><food><name>Strawberry Belgian Waffles</name> <price>$7.95</price><description>light Belgian waffles covered with strawberries and whipped cream</description><calories>900</calories></food><food><name>Berry-Berry Belgian Waffles</name> <price>$8.95</price><description>light Belgian waffles covered with an assortment of fresh berries and whipped cream</description><calories>900</calories> </food>

(26 May '12, 09:33) aaghili

sorry I'm not sure if I understand. The payload isn't this
>,</name>,</price>,</description>,</calories>,</food>,</name>,</price>,</descr iption>,</calories>,</food>,</name>,</price>,</description>,</calories>,</food>, </name>,</price>,</description>,</calories>,</food>,</name>,</price>,</descripti on>,</calories>,</food&gt I posted what the payload should be below. How do I get that? This is just some closing xml tags. Is this a bug?

(26 May '12, 09:36) aaghili

Extracting the text part does so only for each packet separately. You will have to combine the output to get the whole XML response.

I believe what you are looking for is the output of "Follow TCP stream" (Wireshark GUI). If that's the case, there is a new option in tshark 1.7.x, that does almost the same:

tshark.exe -nr xml.cap -z follow,tcp,ascii,5 -q

This will print all data of TCP stream #5 in ASCII format.

To figure out which stream contains XML data, you can use this command:

tshark.exe -nr xml.cap -R xml -T fields -e tcp.stream

It will print the stream number of every TCP stream that contains XML data.

Unfortunately you cannot do this on the fly, during data capture (-i). So you have to capture the data first and then analyze it.

Furthermore I was not yet able to get the out always in ASCII. Sometimes the output of -z follow,tcp,ascii,n is binary and I don't know why. Maybe it's due to GZIP encoding.

If it's the same for you, I suggest to use tcpflow instead. See my answer for this question:

http://ask.wireshark.org/questions/10023/command-line-option-for-follow-tcp-stream

(26 May '12, 11:00) Kurt Knochner ♦

thank you kurt! I'll try this next week with some of our internal services and let you know. Al

(26 May '12, 15:28) aaghili

O.K.

BTW: If you like my answer, you can select it as the right one. see faq.

(27 May '12, 01:21) Kurt Knochner ♦

This option doesn't work for me. Because the output is not sent to standard out but to a file and I need to sniff on an interface. So if I run the command as such 'tshark -i 2 -z follow,tcp,ascii,1 -q' the ouput is written to std out when I hit ctrl-C (only). Our application needs to read the standard out of tshark on the fly. Another way I could get around this problem is to force tshark to dissect all packets as data packets. So even an XML packet be decoded as a data packet. Is there such an option? I want to run tshark -i 2 -R data and see an XML packet in the output of this command.

(27 May '12, 05:49) aaghili

as I said, you can't use -z follow,tcp,ascii,x while sniffing on an interface, as you can't know in which tcp stream (x) your XML data will occur.

If you need that, you have these options

  1. run wireshark with the options described above (no -z follow,tcp) and re-assemble the dumped data yourself to get the whole XML data.

  2. Use libpcap in your own program and extract whatever you need. However, even then you will have to re-assemble several packets to get the whole XML data, and that's not really easier than using tshark.

You could tell us more what you want to achieve. Maybe there is another way to achieve that.

(27 May '12, 12:14) Kurt Knochner ♦

If wanted to use option 1. how would I run to get all the packets to re-assemble. If I run the following tshark.exe -i 2 -T fields -e frame.number -e frame.time -e ip.src -e ip.dst -e data -e text port 80 i always get this </name>,</price>,</description>,</calories>,</food>,</name>,</price>,</descr iption>,</calories>,</food>,</name>t; and not the whole payload. I need to see the whole payload to re-assemble. Basically what I want is to see the whole XML payload either in text/ or hex. I don't mind re-assembling packets.

(27 May '12, 12:46) aaghili

O.K. it looks like -e text does not work properly if the data is GZIP compressed. In that case only part of the data will be printed in ASCII. I will check the code and see if I find any problem (no promisses). If the whole payload in HEX is O.K. for you, you could use option -x:

tshark.exe -i 2 -x port 80

Then extract the data from the output. This works on my system. If the content is compressed, tshark will first print the compressed content and then the "Uncompressed entity body".

Option -V works as well. It's more data to parse, but possibly easier to parse.

You could even try this:

`tshark -i 2 -V -T pdml

Then look for 'field name="xml.tag"'

(27 May '12, 13:10) Kurt Knochner ♦

I check the options you mentioned. if the -T Fields option would give me the whole packet that would have worked best. I don't think this is gzip issue. I tried this with other services and I was getting the same result. It seems like its a bug. Because the only things it prints is the end xml tags only like this. </food>. not the openning tag or the actual body of the element but only the end element.
BTW the reason the -T fields would work best is that the output is on a single line. If I do -R xml then the output comes out in many lines so its much harder to process

(27 May '12, 14:29) aaghili

ok I checked these options and non will be an advantage over just running it like this. tshark -i 2 -R xml -V port 80. Again the problem is that when this is sent to standard out it puts each tag as a separate line. So a large xml payload could have 30000 lines that we need to process. The nice thing about the -T fields option (if it worked) would be that the payload is sent to standard out in a single line no matter how large it is. Much easier to process with better performance.

(29 May '12, 09:32) aaghili

Can you tell me where in the code I can look to find out about this issue? basically I'd like to see the where the ouput of this command is generated.

tshark.exe -i 3 -T fields -e frame.number -e frame.time -e ip.src -e ip.dst -e data -e text port 80

Thanks

(01 Jun '12, 07:40) aaghili

tshark.c:process_packet() -> print_packet() -> print.c:proto_tree_write_fields()

(01 Jun '12, 08:06) Kurt Knochner ♦
showing 5 of 20 show 15 more comments

0

O.K. here is (probably) my last attempt for a solution/explanation ;-)

If you run the following command, you will see the XML tags and the XML data.

tshark.exe -r xml_sample.cap -T fields -R xml -e frame.number -e xml.tag -e xml.cdata -E separator=;

HOWEVER, there is no field xml.data to print the whole XML structure. You are free to extend wireshark/tshark with such a field.

The alternative would be -e text (the 'content-encoded entity body') of the http response.

tshark.exe -r xml_sample.cap -T fields -R xml -frame.number -e text --E separator=;

HOWEVER this does NOT print the full HTTP response. I'm not sure if this works as designed, or if it is a bug (gzip encoding). Please open a bug report, if you think this is a bug.

Then, the above command (-R xml) works ONLY while reading from a file, it does NOT work while reading from an interface (-i), at least not on my test system. So, your attempt to collect the whole XML data structure in one line, on-the-fly, will not work.

Look at the man page for a possible explanation for this (option -R):

Cause the specified filter (which uses the syntax of read/display
filters, rather than that of capture filters) to be applied before
printing a decoded form of packets or writing packets to a file;
packets not matching the filter are discarded rather than being printed
or written.

If the packets that did not match the filter "xml" are dropped, then tshark is probably not able to build the required internal state to collect the whole xml data.

Can someone with better knowldge about the internals, please comment on this?

Regards
Kurt

answered 01 Jun '12, 09:07

Kurt%20Knochner's gravatar image

Kurt Knochner ♦
24.8k1039237
accept rate: 15%

edited 01 Jun '12, 09:14