This is a static archive of our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

tshark -z follow stops after missing package

0

I analyze .pcap and .cap files using tshark's -z follow option. This works mighty fine on most of the streams, however in some streams I get there can be one or more missing/broken packages. In this case the -z option prints only the content up to the first missing/damaged package and everything after that is skipped. How can I get tshark to continue after the missing packets?

UPDATE: As it seems tcpflow and all the other tools I tested face the same problem as tshark... Is there another way to get the tcp data then "-z follow"? Where is this option implemented in the Wireshark source?

asked 12 Feb '14, 06:03

tonivalac's gravatar image

tonivalac
1115
accept rate: 100%

edited 17 Feb '14, 02:15

Some questions:

  • what is your OS and OS version of the Wireshark system?
  • what is your Wireshark version?
  • how do you capture the traffic (with Wireshark, or other tools)?
  • is it possible to post a sample capture file somewhere (google drive, dropbox, cloudshark.org)?
(12 Feb '14, 06:22) Kurt Knochner ♦

I need something that runs on Windows and Linux, the capture is done with a different tool and I am really sorry, but for various reasons I can not post a sample capture file.

To make my point clearer: I have running code and I know the flows contain more data then the -z follow option (or Wiresharks follow tcp stream) display. This is caused by missing or malformed packages. As far as I know tshark/wireshark is configured to stop extracting the data from a flow on such packages by default.

What I would like best would be a solution where I can keep using tshark -z follow, but configure it to skip the missing/malformed packages and reconstruct all the correct data in a flow

(12 Feb '14, 06:42) tonivalac

what is your Wireshark version?

(12 Feb '14, 06:53) Kurt Knochner ♦

On Windows it's 1.10.1, but I think it has nothing to do with the version, but is the same for all versions since the -z follow option was implemented

(12 Feb '14, 06:55) tonivalac

That version should not show the effect you are describing (at least it does not in my tests), that's why I asked for a capture file sample.

(12 Feb '14, 06:56) Kurt Knochner ♦

BTW: do you need the ascii/hex output of tshark -z follow or just one new capture file per tcp stream of the original capture file?

(12 Feb '14, 06:58) Kurt Knochner ♦

Hmm... actually I found quite a lot people stating it can't handle missing packets and as I said posting pcaps is unfortunately not an option

(12 Feb '14, 07:01) tonivalac

it can't handle missing packets

it depends on your definition on "can't handle" and "missing packets".

BTW: do you need the ascii/hex output of tshark -z follow or just one new capture file per tcp stream of the original capture file?

(12 Feb '14, 07:04) Kurt Knochner ♦

I need the ascii output and can't handle means tshark knows there's more data in the flow, because I can see the data if I select the individual package in wireshark, but if I use -s follow or follow tcp stream on a package after a missing one, the ascii/hex/raw output stops where the first missing package should have been

(12 Feb '14, 07:15) tonivalac
showing 5 of 9 show 4 more comments

2 Answers:

0

I now found a way to do it that works with almost any protocol. Here the tcp example:

tshark -e tcp.stream eq [stream number] -e data -T fields -r [path to pcap]

This writes a lot of hex numbers to stdout, which can be easily converted to ASCII using for example python's binascii.unhexlify

answered 26 Feb '14, 03:48

tonivalac's gravatar image

tonivalac
1115
accept rate: 100%

edited 27 Feb '14, 01:12

1

but if I use -s follow or follow tcp stream on a package after a missing one, the ascii/hex/raw output stops where the first missing package should have been

O.K. then you cannot use tshark and you should take a look at other tools, like one of the following

https://isc.sans.edu/diary/Tools+for+extracting+files+from+pcaps/6961

Regards
Kurt

answered 12 Feb '14, 07:29

Kurt%20Knochner's gravatar image

Kurt Knochner ♦
24.8k1039237
accept rate: 15%

Sad but true. Thx for the link.

(13 Feb '14, 00:12) tonivalac

Hint: If a supplied answer resolves your question can you please "accept" it by clicking the checkmark icon next to it. This highlights good answers for the benefit of subsequent users with the same or similar questions.

(13 Feb '14, 00:18) Kurt Knochner ♦

Ok. Unfortunately no tools supports the data link type I use... now I'm wondering why tshark can't just skip those packages and continue with resembling the stream. Any ideas on how to make this possible (for example by changing some lines in the tshark source code)?

(13 Feb '14, 03:04) tonivalac

Can you change the DLT with editcap?

(13 Feb '14, 04:13) Kurt Knochner ♦

"-z follow"? Where is this option implemented in the Wireshark source?

ui/cli/tap-follow.c

BTW: please don't update your original question with new sub-questions, as it's extremely hard to follow the 'flow' that way.

(17 Feb '14, 04:04) Kurt Knochner ♦

++ UPDATE ++

I did some tests with capture files that contain Retransmissions, missing frames, DUP ACK, etc.

In all cases the function "Follow TCP stream" did not stop before the end of the conversation, as far as I can tell (Wireshark 1.10.5 and 1.11.2).

Example files:

Running the following tshark command on both files,

tshark -nr ask.wireshark.org.with_get.pcap -q -z follow,tcp,ascii,0 > output1.txt
tshark -nr ask.wireshark.org.no_get.pcap -q -z follow,tcp,ascii,0 > output2.txt

gives almost the same output. Obviously the HTTP GET request is missing in the 2nd output file.

but for various reasons I can not post a sample capture file.

Looks like the problem you are describing is not easy to reproduce. Maybe it's related to the content ('strange characters' in the output) or the protocol being used in your capture file, or anything else...

Unless you can reproduce the problem with a capture file you can share, it's kind of difficult to help you,

(17 Feb '14, 07:38) Kurt Knochner ♦

I have missing frames similar to your "TCP previous segment not captured" and even in your sample, the ascii output is cut after this frame on my system, even with Wireshark 1.10.5 and with 1.11.2 tshark -q -z follow,tcp,ascii,0 is not working at all

EDIT: Ok, somehow I got it working now under 1.10.5 with your pcap... however not with mine. I guess I'll write a script for analysing pcaps myself. Thx anyway

(17 Feb '14, 23:08) tonivalac

So, what did you change to make it work with my pcap??

(17 Feb '14, 23:59) Kurt Knochner ♦

I reinstalled Version 1.10.5 after having installed 1.11.2

(18 Feb '14, 06:23) tonivalac

strange. It works with both versions on my system.

Anyway, as I said: unless you can provide a sample capture that shows the problem (I was unable to find/create one), it's kind of difficult to help you.

(18 Feb '14, 06:35) Kurt Knochner ♦

Ok. I wrote a script now, which analyzes the pcaps for me, so I guess we can close this now and maybe in a future tshark Version my misterious problem will disappear all by itself...

(19 Feb '14, 00:18) tonivalac

I just found the reason, why "follow tcp stream" aborts. My files have only one direction. If I use "ip.src eq 192.168.158.139" as filter in your example files and then use "Export Specified Packets", I get the same error I have in my pcaps.

Is there any way to easily fix this? Or can I use some other tshark command to get the ascii data of the tcp stream? I keep thinking that this really shouldn't be too hard for Wireshark "sees" all the right packets using "tcp.stream eq 0" as display filter

EDIT: Just realized that there IS no data in the direction containing the error. Could you please provide a pcap where there is a "TCP Previous segment not captured" error in a conversation containing real ascii data, which was captured unidirectional?

(21 Feb '14, 01:22) tonivalac

I know deleted one packet in Kurt's "no_get.pcap" and confirmed my suspicion: A missing packet in a bidirectional capture is no problem for "-z follow" and "follow tcp stream". The same packet in the same conversation in a unidirectional capture breaks "-z follow" and "follow tcp stream"

However in both uni- and bidirectional captures a display filter "tcp.stream eq <number>" shows all the captured packets, not only those before the missing one. I consider this a bug in tshark and would really appreciate if anybody could either help me solving this problem or provide a good work around (preferably using tshark)

(21 Feb '14, 03:35) tonivalac

@tonivalac

Your "answer" has been converted to a comment as that's how this site works. Please read the FAQ for more information.

(21 Feb '14, 05:07) grahamb ♦

I'm not sure if this is a bug, as a unidirectional conversation is not a full TCP conversation. Of course you can file an enhancement bug at https://bugs.wireshark.org to add support for unidirectional filtered capture files.

(21 Feb '14, 08:50) Kurt Knochner ♦

Well, I think it is, for wireshark "knows" the frames after the missing one belong to the same tcp stream and if it simply ignores the missing frame in "follow tcp stream" with both directions, it should do the same for one direction. However I still don't know how to extract the ascii data content from my pcap...

(23 Feb '14, 23:59) tonivalac

it should do the same for one direction.

as that functionality is apparently not implemented, you should file an enhancement bug, as explained in my last comment.

(24 Feb '14, 01:32) Kurt Knochner ♦

Thx Kurt, I did that of course, but this still does not solve my original problem.

(24 Feb '14, 01:56) tonivalac

I know. So, let's go back to your problem. Maybe there is totally different solution. So, here are some more questions.

  • why do you filter the conversation to have only one direction?
  • did you try the other tools I mentioned in my answer with a bi-directional capture file? Some of the tools will split the stream anyways into c->s and s->c.
  • what exactly do you need from the TCP stream? Is it the full 'text', or just some 'keywords'?
  • what kind of protocol is this (http, etc.?)
(24 Feb '14, 05:10) Kurt Knochner ♦

Actually I do not really filter the conversation, I only capture one direction and this is something I can't and won't change.

I need the full 'text' and the protocol can be almost anything

(24 Feb '14, 22:46) tonivalac

Any protocol? In many protocols there is no readable 'text'. So, what are you actually trying to do? Maybe there is a totally different solution.

(24 Feb '14, 23:48) Kurt Knochner ♦

I want all ASCII chars in the highest layer of a TCP stream. Other protocols should work, too, but I guess I could spare that

(24 Feb '14, 23:59) tonivalac

O.K. let me ask the other way round. How much sense does it make to look for ASCII text in any TCP protocol?

So, again: what are you actually trying to do? Maybe there is a totally different solution.

(25 Feb '14, 00:14) Kurt Knochner ♦

I already tried that solution, but it is not satisfying. I want the same intelligence in handling the packets as tshark, which is why I'd really like to find a workaround using the current tshark version.

What am I trying to do? I guess I told you what I am trying to do... at least I tried. What did I forget to mention? What exactly was vague?

(25 Feb '14, 00:32) tonivalac

O.K. here comes my last attempt ;-))

Why do you need the ASCII text of TCP connections, regardless of the protocol? What are you going to do with that data?

Build some kind of IDS (Intrusion detection system) or DLP (data leakage 'prevention' - actually leakage detection)?

If yes, neither tshark nor Wireshark is the right tool for you, for several reasons.

If no, please add more details if you want any further help!!!

(25 Feb '14, 09:56) Kurt Knochner ♦
showing 5 of 25 show 20 more comments