This is a static archive of our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

Select Ip source addresses with 100 packets or more and save to another file

0

I have a large pcap file that I split in several smaller files using editcap. I sorted each file by IP source address, (using |sort -k 3 command), but now I need to select only those IP source addresses with 100 packets or more, and then write only those addresses to another file for further filtering. I need to do that from command line because I have to write a bash script to do that for all smaller files resulting from the split (about 800 files). Help is greatly appreciated.

asked 23 Nov '16, 20:03

MaryR's gravatar image

MaryR
26558
accept rate: 0%

In what form is your packet data? From the description I assume it's not in PCAP files. Would that be CSV files instead?

(24 Nov '16, 01:20) Jaap ♦

I forgot to mention it is a pcap file. Thanks.

(24 Nov '16, 06:51) MaryR

One Answer:

2

You could script something like this to create a list of all IP addresses with at least 100 packets in a single small file:

for file in `ls -1 <all-small-files>`
do
    tshark -r $file -T fields -e ip.src | sort | uniq -c | awk '$1 >= 100 {print $2}' 
done | sort -u > ip-with-100-packets.txt

Or you could use tcpdump on the large file (as it does not keep state and therefor can handle the big file without running out of memory). :

tcpdump -n -r netflix.pcapng  | cut -d ' ' -f 3 | sed -Ee 's/\.[0-9]+$//' | sort | uniq -c | awk '$1 >= 100 {print $2}' > ip-with-100-packets.txt

This will give you all ip's with at least 100 packets in the original large file.

answered 24 Nov '16, 02:12

SYN-bit's gravatar image

SYN-bit ♦♦
17.1k957245
accept rate: 20%

Thank you so much for these solutions SYN-bit. The second solution in my computer is taking 15 min + (still running), so perhaps file is too large (822 MB). But for the first solution, is there a way to still keep the time stamp (2nd field) and the TCP/UDP (5th field) fields, in addition to the IP source address field in the resulting file? Because I still have to verify that each of those 100+packet flows have a duration of at least 60 seconds, and I have to indicate the type of DDoS flooding (TCP Flooding, UDP Flooding or ICMP Flooding). Your help very much appreciated.

(24 Nov '16, 07:39) MaryR

The second solution works well, I just added ">>" so the output file is not overwritten but appended instead. As I mentioned before, I would like to keep the other fields (timestamp and TCP/UPD) on the final output file, is that possible? Thanks again.

(24 Nov '16, 22:10) MaryR