Utility to sort pcap or pcapng capture file based on timestamp?

Question

Hello: I am using a PC with two NICs to capture both directions of traffic using a Gigamon Copper tap. Unfortunately it appears that one of the NICs is faster than the other in writing to the unified capture file and I have a large number of "TCP ACKed unseen segment" tagged packets in the file. If I sort by time stamp the packets are in the correct order but Wireshark appears to interpret based on the packet number rather than the time stamp.

I was hoping that mergecap would give me a workaround for this situation, but mergecap assumes that all packets within a capture file are already in correct order. I have also tried editcap's -S function to adjust the timestamps.

Any ideas? Is there a utility that will force a resort of packets by time stamp?

Thank you,

Brad

Accepted Answer

3

The development version of Wireshark has a utility called 'reordercap' which does just this. From the man page:

Reordercap is a program that reads an input capture file and rewrites the frames to an output capture file, but with the frames sorted by increasing timestamp.

This functionality may be useful when capture files have been created by combining frames from more than one well-synchronised source, but the frames have not been combined in strict time order.

You might want to pick up a development build of Wireshark to check it out.

answered 01 Feb ‘13, 06:41

JeffMorriss ♦
6.2k●5●72
accept rate: 27%

Nice, I didn’t know about that… now I can compare what my tool does and see if I do it right :-)

BTW I just looked at the code and if I read it correctly (which may not be the case as my C skills are basic at best) reordercap will read ALL frames, sorting them in the next step, and then writing them back out again. I’m not sure how well that will work with huge files. I guess frames are kept in memory before sorting.

I do it slightly different by reading frames in chunks of, lets say, 10000 frames and asynchronously writing them back out in a sorted order. File Size doesn’t matter.

(01 Feb ‘13, 09:53) Jasper ♦♦

1

There were several versions of reordercap, using different data structures / sorting algorithms. The first one used a sorted list, with an option to limit its size - i.e. if the unsorted list reached the max it would write the earliest entry seen to the output. Like the current version, it only stored header information, and would go back to reload the frame to write it. The current version has been used with 4GB files, but unless you have tiny frames the memory usage will be a small fraction of the file size. Because it uses wiretap it will work with any supported capture file type.

(01 Feb ‘13, 11:30) MartinM

Ah, okay, that makes sense :)

(01 Feb ‘13, 13:59) Jasper ♦♦

Jeff: This worked great! I pulled the portable version in order to have a parallel install with the stable version and used the reordercap utility on my problematic file. All of the “ACKed unseen segment” alerts vanished.

Thank you!

Brad

(04 Feb ‘13, 13:00) Brad Steffen

reordercap - in release 1.10.1 - only seems to work if the out-of-order frames have timestamps that are also out-of-order.

typically the timestamp is added at the time of capture, so out-of-order frames will be timestamped in the order they were captured.

A better solution would be to reorder the frames using the tcp.seq and tcp.nxtseq numbers

(03 Sep ‘13, 06:04) Steveep

A better solution would be to reorder the frames using the tcp.seq and tcp.nxtseq numbers

this will modify and thus (possibly) invalidate the original ‘tcp flow’. Out-of-order frames will be sorted in a different order than they appeared on the net, which makes an analysis hard or even impossible, because you will see answers from one side, that you can’t explain/understand without knowledge about the out-of-order frames.

(03 Sep ‘13, 07:55) Kurt Knochner ♦

showing 5 of 6 show 1 more comments

Answer 2

Yes, there is a utility that can do this, but it is not publicly available yet. I'm currently writing a windows based tool for working with pcapng files that will offer a couple of functions to help with the daily work of analysts.

Right now, I'm spending most time on coding trace file anonymization functionality which is more comfortable than what is already out there. But in the process of building my data processing objects I implemented a function that can read a pcapng file and write it back to disk with all frames sorted by timestamp (which, as you already mentioned, happens a lot when capturing on multiple NICs at the same time). That sorting function worked well in a quick test I did last week, but I haven't tested it on a larger scale, and the maximum file size limit right now is still 2GB per file. The current plan is to demo/release the first version of that tool in a session at this year's Sharkfest conference (the web pages are not yet updated for 2013).