This is a static archive of our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

Is it possible to use a blacklist.txt file as an input list for a wireshark | tshark displayfilter?

0

Is it possible to use a blacklist.txt file as an input address list for a wireshark or tshark display filter? My current blacklist of 4000 entries errors out in the display filter box ( syntax is ok , i wonder what max number of entries is ;-) ). I'd be very happy if something exists along the lines of:

for file in *.pkt
tshark.exe -r $file -Y "ip.addr == blacklist.txt" -w $file-toblacklist.pcap
done

asked 25 Aug '15, 07:40

Marc's gravatar image

Marc
147101316
accept rate: 27%

Right, the limit of the Wireshark display filter field seems to be 64K alt text

which leaves room for about 2290 ip addresses in one filter pass ... so for now i guess i'll filter all files twice .. :-(

(26 Aug '15, 00:04) Marc

So much for trying that in cygwin, argument list is too long ..

alt text

(26 Aug '15, 01:15) Marc

Do you need only a list of non-blacklisted IP addresses in the capture files, or do you want to remove frames from/to the blacklisted IPs in all capture files?

(29 Aug '15, 03:14) Kurt Knochner ♦

The correlation i need is in which files do what blacklisted ip adresses occur?

(30 Aug '15, 01:54) Marc

my perl script does that. It does not output a simple list of files, it writes all frames from/to blacklisted IPs to a new file called *_blacklist.pcap. So, actually you get what you want (a list of files that contain blacklisted IPs), but it also gives you the whole communication from/to blacklisted IPs.

(30 Aug '15, 02:57) Kurt Knochner ♦

4 Answers:

1

One option would be:

for ip in $(cat blacklist.txt); do tshark -r file.pcapng -Y ip.addr==$ip; done

answered 26 Aug '15, 17:24

Roland's gravatar image

Roland
7642415
accept rate: 13%

Thanks , that will do for the tshark part of my Q ..although i guess this will take 1 ip address at a time, filter the pcap file and then hop to the next , for 4000 ip adresses X 2183 tracefiles of 262 MB it'll be a long process...

(28 Aug '15, 23:56) Marc

In your question there was no mention of 2k trace files. The command in your last comment should be:

for IP in $(cat blacklist.txt); do grep "$IP" *.txt >> BADIPS.csv; done

(30 Aug '15, 02:51) Roland

1

You could write a Lua plugin script to do it, running inside tshark or Wireshark. You can pass the filename(s) of your blacklist to the Lua plugin using the command line, and have the Lua plugin open the blacklist file(s) and build a match table, and have the plugin create either a post-dissector or Listener tap, which will get invoked for every packet letting you check the source and dest IP addresses against the match table; and if it matches then have the plugin either (1) add a field called "blacklisted" which you can then use as your actual display filter for tshark/wireshark, or (2) have the plugin save the blacklisted packet to a new pcap file directly.

answered 26 Aug '15, 19:02

Hadriel's gravatar image

Hadriel
2.7k2939
accept rate: 18%

Ok, sweet , have to learn how to do that then ;-) But the extra possibilities do appeal to me! , Thanks!

(28 Aug '15, 23:58) Marc

1

Based on the information provided so far, I conclude, that you want to write the packets to/from blacklisted IP addresses to a new pcap file.

If so, here is my simple Perl example. Save the code to a file and run it like this:

perl extract_blacklist_frames.pl blacklist.txt capturefile1.pcap

This will create a new pcap file called: capturefile1.pcap_blacklist.pcap (I did not invest any time to create a 'pretty' new name. I leave that up to you).

If you want to process several files, run it in a loop on Unix/BSD like systems with a bash

for name in *.pcap
do 
echo processing file: $name
perl extract_blacklist_frames.pl blacklist.txt $name
done 

Blacklist file: one IP per line!

1.2.3.4
10.10.10.10
2.3.4.5

Perl code:

use Net::Pcap qw(:functions);
use NetPacket::Ethernet qw(:types);
use NetPacket::IP qw(:protos);

use warnings; use strict;

my %blacklist;

my $blacklist_file = $ARGV[0] || die ("ERROR: please specifiy the name of the blacklist file\n"); my $pcap_file = $ARGV[1] || die ("ERROR: please specifiy the pcap file name\n"); my $pcap_file_blacklist = $pcap_file . "_blacklist.pcap";

my $pcap_read; my $pcap_write;

read_blacklist(); process_pcap_file();

sub read_blacklist {

open(BLACKLIST, $blacklist_file) || die ("FATAL: cannot open blacklist file: $blacklist_file\n");
while (<BLACKLIST>) {
    chomp;
    $blacklist{$_} = 1;
}
close (BLACKLIST);

}

sub process_pcap_file {

my $err;
my $maxpkts;

$pcap_read = Net::Pcap::open_offline($pcap_file, \$err) || die("FATAL: cannot read pcap file $pcap_file: $err\n");
$pcap_write = Net::Pcap::dump_open($pcap_read, $pcap_file_blacklist) || die("FATAL: cannot write pcap file $pcap_file_blacklist: $err\n");

Net::Pcap::loop($pcap_read, $maxpkts, \&process_packet, '');

Net::Pcap::close($pcap_read);
Net::Pcap::dump_close($pcap_write);

}

sub process_packet { my ($user_data, $header, $packet) = @_;

my $ip = NetPacket::IP->decode(NetPacket::Ethernet::strip($packet));
my $src_ip = $ip->{src_ip};
my $dst_ip = $ip->{dest_ip};

if (exists $blacklist{$src_ip} or exists $blacklist{$dst_ip}) {
    print "Blacklist IP found, writing packet\n";
    Net::Pcap::dump($pcap_write, $header, $packet);
}

}

HINT: The Perl script will be able to read pcap-ng files only if the libpcap version on your system is able to read pcap-ng, otherwise the script will throw an error! If that happens, either install the latest libpcap or convert the capture files to pcap style with editcap.

Regards
Kurt

answered 29 Aug ‘15, 07:52

Kurt%20Knochner's gravatar image

Kurt Knochner ♦
24.8k1039237
accept rate: 15%

edited 29 Aug ‘15, 08:08

Hi Kurt, cool , i’ll give your Perl script a try too! Sofar i’ve come up with:

for file in *.pkt do tshark.exe -r $file -q -z conv,ip >> $file-IPsummary.txt;done

for IP in cat blacklist.txt; do grep "$IP" *.txt >> BADIPS.csv;done

although only the first part works at the moment .. :-)

(30 Aug ‘15, 01:51) Marc

btw i know this should be a comment but have no clue on how to post code in a comment ..

(30 Aug ‘15, 01:52) Marc

I converted your answer to a comment. What I do, if I need formatting in a comment: I write the comment as an answer and if it looks O.K. in the preview I copy the text in the edit window and paste it into a commment.

(30 Aug ‘15, 02:54) Kurt Knochner ♦

Hi Kurt, yep, that works! Thanks for the script. I will try it out in due time , for now i stuck with the path of getting the info from the t-shark summary command and then processing it afterwards, like so ( thanks to Roland for cleaning up the second line too)

for file in *.pkt do tshark.exe -r $file -q -z conv,ip >> $file-IPsummary.txt;done
for IP in $(cat blacklist.txt); do grep "$IP" *.txt >> BADIPS.csv; done

only reason being speed at the moment ;-)

(01 Sep '15, 23:54) Marc

0

Here's a quick Python script for you. It outputs any IP addresses in the .pcap which match those in the blacklist file.

Requires Python 3. Save this as a python file (.py) and run it (./filename.py).

Change these in the code to the files you're analysing: [.pcap file = packets.pcap] [blacklist file = blklstfile]

#!/usr/bin/python
import os
os.system("tshark -r packet.pcap -T fields -e ip.src | sort | uniq > temp")
blklst = [line.rstrip("\n") for line in open("blklstfile")]
temp = open("temp", "r")
for blkip in temp.readlines():
 blkip = blkip.rstrip("\n")
 if blkip == "#":
  continue
 check = blkip
 if check!="" and check in blklst:
  print("Blacklisted IP found:" + check)

Not quite what you're asking for but it will flag any blacklisted IP addresses if they appear in the PCAP file.

answered 27 Aug '15, 13:52

tbm's gravatar image

tbm
29116
accept rate: 0%

edited 27 Aug '15, 14:43

Right! Same here , looking like exactly what i want and all of these are different paths to get to the same answers: do i have any blacklisted ip's in a load of pcap files? It will be a bit of a learning curve do it either with Lua or Python but i will give it a whirl! However i do like the idea of letting the shark create a 'ip summary' first then automate some unix search power on those lists .. .. Something along the lines of "tshark -r <pcapfile> -q -z conv,ip > pcapfile_ip_summary.csv" then search those "for ip in $(cat blacklist.txt); do ... .

(29 Aug '15, 00:03) Marc