bulk extraction of UDP payload data

Question

Hi All,

In brief, what I have:

50GB (100 x 512MB) pcap files captured with dumpcap containing multicast UDP data (mixed target ports).
Matlab scripts dealing with the raw binary payload data for a selection of ports.

What I want to do:

For each of the 100 pcap files, extract raw UDP binary PAYLOAD-ONLY data sent to a specific port X.
Repeat process for each port of interest (eg if I had 7 ports of interest, I'd end up with 700 files containing payload data)

Essentially it is the same conversion performed by Wiresharks "Follow UDP Stream", but the data volume is too high to use that feature.

I noticed a couple of other threads dealing with this or similar issues:

https://ask.wireshark.org/questions/38998/automating-extraction-of-udp-payload-from-pcap-file/

https://ask.wireshark.org/questions/15374/dump-raw-packet-data-field-only/

but I feel I am missing something, it seems the data has to go to ASCII first? Rather than straight to binary. With such large volumes of data, I was hoping there might be a perl or python trick out there to help batch this job. I'm not that familiar with PERL, but I have started looking into the Net::Pcap libraries.

I have no use for the additional packet info/stats collected by dumpcap, so in hindsight I probably would have been better off coding up a simple bit of software to receive and record the UDP data of interest directly, but it's a bit late for that now.

Any further tips greatly appreciated.

Answer 1

Please take a look at my perl code in a similar question. Maybe you can use it as a starting point.

https://ask.wireshark.org/questions/15928/how-to-export-hex-and-timestamp

Instead of printing the hex representation of the UDP payload, you could write it to a binary file.

++UPDATE++

So, the code could look like this (works on my systems).

BTW: Net::Pcap will be able to read the pcap format by default. It will be able to alo read pcap-ng, if the version of libpacap on your system does support it, otherwise you will get an error message ("FATAL: cannot open 47183.pcapng -> ERROR: bad dump file format")

#!/usr/bin/perl
use warnings;
use strict;
use Net::Pcap;
use NetPacket::Ethernet qw(eth_strip);
use NetPacket::IP qw(:ALL);
use NetPacket::UDP;
use POSIX qw(strftime);
my $pcap_file = $ARGV[0];
my $port_list = {
53 => 1,
5353 => 1,
1900 => 1
};
my $filehandles;
my $error;
my $pcap = Net::Pcap::open_offline($pcap_file, $error) or die("FATAL: cannot open $pcap_file -> ERROR: $error\n");
Net::Pcap::loop($pcap, -1, &amp;process_packet, '');
Net::Pcap::close($pcap);
sub process_packet {

my ($user_data,$header, $packet) = @_;
my $ip = NetPacket::IP-&gt;decode(eth_strip($packet));
my $src = $ip-&gt;{src_ip};
my $dst = $ip-&gt;{dest_ip};

if ($ip-&gt;{proto} == IP_PROTO_UDP) {

    my $udp = NetPacket::UDP-&gt;decode($ip-&gt;{data});
    my $udp_sport = $udp-&gt;{src_port};
    my $udp_dport = $udp-&gt;{dest_port};

    if (not exists $port_list-&gt;{$udp_dport}) {
        return;
    }

    my $session = &quot;${src}_${udp_sport}__${dst}_${udp_dport}&quot;;

    write_payload($session, $udp-&gt;{data});
}

}
sub write_payload {
my $session = shift;
my $data = shift;
my $fh;

if (exists $filehandles-&gt;{$session}) {
    $fh = $filehandles-&gt;{$session};
} else { 
    my $filename = $pcap_file . &quot;__&quot; . $session . &quot;.payload&quot;;
    open($fh, &quot;&gt;&gt;:raw&quot;, $filename) || die(&quot;FATAL: cannot write to file $filename\n&quot;);
    $filehandles-&gt;{$session} = $fh;
}

print $fh $data;

}

My script writes only the payload of one direction to the file (it looks for destination port), as it won’t make much sense to mix data of both directions, packet by packet, into the binary output file. If you need something different, please adjust the script to your needs.

Regards
Kurt