This is a static archive of our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

What' s the best way to split tcp streams on large pcap files?

0

Hi,

I'm trying to dump all tcp streams from a large pcap file into separate files. I've used the lua interface for doing it at the best way possible. But the problem is that i reach the error "Too many open files" on the operating system because not all flow my and with a FIN and there is no way to acknowledge that the packet is the last packet on a tcp strem. Thx in advance, Leonardo

asked 10 Jan '15, 13:18

singletron's gravatar image

singletron
11●1●1●2
accept rate: 0%

edited 11 Jan '15, 09:44

Hadriel's gravatar image

Hadriel
2.7k●2●9●39

1

Perhaps there needs to be a way in Lua to open a "Dumper" file in append mode, so you could open and close the appropriate Dumper file on each packet to avoid running out of file handles. (or keep up to a few hundred open at any given time, and close them when you get too many)

(11 Jan '15, 09:36) Hadriel
1

Added enhancement bug 10847.

(11 Jan '15, 09:43) Hadriel

Dumper in append mode would really be a plus and solve this issue. What i'm trying now is to load all the tcp streams into memory and dumping them after into files. But this is a really memory hungry process. What i'm finding out also is that the ByteArray class only accepts hex string as input and that is not compatible with the Tvb raw output also. Will try the BitOp lua module to see if it speeds up the process. But thanks for the feedback!

(11 Jan '15, 14:29) singletron
1

You can already create a ByteArray from a Tvb - just create a TvbRange of the Tvb, and call bytes() of the TvbRange - that returns a ByteArray. In other words:

-- assuming myTvb is a Tvb object
local barray = myTvb():bytes()

– or this way
local barray = myTvb:range():bytes()


– or this longer way
local tvbr = myTvb:range()
local barray = tvbr:bytes()
(11 Jan ‘15, 16:29) Hadriel

Wow that is cool i think this is going to make my script work in a feasible speed! Many thanks!

(11 Jan ‘15, 16:38) singletron

2 Answers:

0

I think you might want to use a tool other than Wireshark for that, because as you noticed you'll run into the file handle problem when trying to separate a large number of streams. Right now I'd recommend TCPFlow, which should help you getting your streams.

answered 11 Jan '15, 06:02

Jasper's gravatar image

Jasper ♦♦
23.8k●5●51●284
accept rate: 18%

The problem with TCPFlow is that it creates streams into the two way separate flows. I would have to write some extra code to put them together into a single file. Thanks!

(11 Jan '15, 14:31) singletron

And of course it doesn't have the robustness of wireshark if you want to deal with more complex filters.

(11 Jan '15, 14:32) singletron
1

You could merge the two flows with mergecap, by timestamp. That can be scripted easily.

Filtering could be done in Wireshark before exporting the single flows. So that should not be a problem.

(11 Jan '15, 14:34) Jasper ♦♦

0

I developed a tool that fits exactly to your needs: PcapSplitter. There's also a compiled version for several OS's here. It can split pcap files into streams and it doesn't have a "too many open files" problem as it closes and reopens files during its run. You should run it in the following way to achieve what you need:

./PcapSplitter -f /path/to/your/file.pcap -o /output/dir -m connection

answered 23 Jul '16, 12:36

seladb's gravatar image

seladb
1●1
accept rate: 0%