Split pcap file into smaller pcap file (according to tcp flow)

Question

I need to find a way to split a large pcap file into separated pcap files. What I want to find is a application like Splitcap but I need a application which runs on Linux. Tcpflow or Tcptrace don't generate pcap file as their output. The output pcap file should contains a tcp flow.

If there's an application, please let me know. Split pcap file using tshark will be very helpful for me.

Answer 1

1

tshark can do that.

tshark -nr input.cap -R "tcp.stream eq 1" -w stream_1.cap

Regards
Kurt

answered 07 Dec '12, 08:31

Kurt Knochner ♦
24.8k●10●39●237
accept rate: 15%

Thanks for the comment, Kurt. But my input file contains more than one millinon flows. Is there any other options to do this?

(07 Dec '12, 08:33) fates

1

And you are asking for what? Having 1 million files, one for each stream?

If so, you can run tshark in a loop and use the loop counter in the stream filter and the output file name.

See the following question:

http://ask.wireshark.org/questions/4677/easy-way-to-save-tcp-streams

(07 Dec '12, 08:41) Kurt Knochner ♦

A easier/faster method would be this python script:

http://corelabs.coresecurity.com/index.php?module=Wiki&action=attachment&type=tool&page=Impacket&file=split.py

http://corelabs.coresecurity.com/index.php?module=Wiki&action=view&type=tool&name=Impacket

The script needs pcapy. You can install pcapy on Ubuntu like this:

apt-get install python-pcapy

(07 Dec '12, 08:57) Kurt Knochner ♦

Thanks, Kurt! By the way, I've already tried the python script. :) But this script uses "Impacket" package and this package cannot handle corrupted packets. This is why I'm trying to find other solutions.

(07 Dec '12, 09:51) fates

well, then use tshark.

(07 Dec '12, 09:59) Kurt Knochner ♦

Answer 2

Separate the packets into flows considering only 4 tuples: source address, source port, dest address, dest port for further analysis.

The packets are saved in the time order without any processing like TCP resembling.

The flow timeout is considered as 64 seconds suggested by CAIDA.

https://github.com/caesar0301/pkt2flow

Answer 3

The tshark scripts didn't finish in 30 minutes on my 4G pcap with about 40 flows. The following finished in about 90 seconds. tshark versus tcpdump?

#!/usr/bin/perl -w
use strict;
use Data::Dumper;
sub mysystem {
my ($s, $donothing) = @_;
chomp $s;
print "$s\n";
if ( defined $donothing ) {
return;
}
my $rv = system "$s > cmd.out";
if ($? == -1) {
die "failed to execute: $!\n";
} elsif ($? & 127) {
die sprintf "child died with signal %d, %s coredump\n",
($? & 127),  ($? & 128) ? 'with' : 'without';
} elsif( $rv ) {
$rv = $rv/256;
die "$s exited with status $rv\n";
}
cat cmd.out;
}
return 4-tuples. the protocol is always tcp.
sub identify_tcp_flows {
my $pcapfn = shift;
my %flows;
open F, "tcpdump -n -r ${pcapfn} tcp |" or die "fozzle";
while (<F>) {
if ( m{
\A
(?<timestamp>
\d{2} :
\d{2} :
\d{2} [.] \d+
) \s+
IP \s+
(?<src_ip>
\S+
)
[.]
(?<src_port>
\d+
) \s+
> \s+
(?<dst_ip>
\S+
)
[.]
(?<dst_port>
\d+ | http
) : \s+
Flags \s+
[[]
(?<flags>
[^]]+
)
[]] , \s+
(
seq \s+ \d+ , \s+
)?
(?<ack>
ack \s+
(?<ackbytes>
\d+
) , \s+
) ?
}xms
) {
if ( ! exists $flows{"$+{dst_ip}:$+{dst_port}-$+{src_ip}:$+{src_port}"} ) {
$flows{"$+{src_ip}:$+{src_port}-$+{dst_ip}:$+{dst_port}"} = {
src_ip => $+{src_ip},
src_port => $+{src_port},
dst_ip => $+{dst_ip},
dst_port => $+{dst_port},
};
}
} else {
warn "couldn't parse $_";
}

}
%flows
}
my $pcapfn = $ARGV[0];
my $r_h_flows = identify_tcp_flows $pcapfn;
for my $f ( keys $r_h_flows ) {
mysystem "tcpdump -n -r ${pcapfn} -w $f.pcap &quot;tcp and host $$r_h_flows{$f}{src_ip} and host $$r_h_flows{$f}{dst_ip} and port $$r_h_flows{$f}{src_port} and port $$r_h_flows{$f}{dst_port} &quot;";
}

Answer 4

You can use PcapSplitter which is part of the PcapPlusPlus suite. It's cross-platform so it can run on both Windows, Linux and Mac OS X. There's also a binary version for several OS's here. It can process large pcap files containing large amount of streams (both TCP and UDP). You should use it as follows:

./PcapSplitter -f /path/to/your/file.pcap -o /output/dir -m connection