This is our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.
1
1

Hello,

I am trying to write a script that will remove duplicate packets (layer 3 and above) by comparing the data in two consecutive packets (or along a sliding window) and if they are identical, it will throw out the duplicates. The packets are not 100% duplicates since the MACs and TTLs are different, but from layer 3 and above they are identical. Therefore, editcap -d won't work, as it requires the MD5 hash to be the same.

The way I would like to do this is to output the data like this:

Mar 5, 2013 19:34:39.738281000 0000 00 15 f1 9c f8 00 00 01 fc 0d 68 c0 81 00 0c 90 ..........h..... 0010 08 10 46 a0 04 62 00 00 00 00 6e 11 05 c5 ac 10 ..E..c....~..... 0020 57 c6 ac 10 82 3e 13 c4 1a 0a 04 4f de 41 52 45 W....>.....O.ARE 0030 47 49 53 54 45 52 20 73 69 70 3a 66 61 70 2e 61 GISTER sip:data. ... 0460 36 2e 31 43 30 2e 36 32 3a 37 36 22 36 3b 6c 63 6.120.62:3626;fd 0470 3e 0d 0a 0d 0a >....

Then I can use a script to check the data for duplicates and use the timestamp at the beginning of the packet to read it back into a pcap file.

The command below summarizes what I have so far. What I need is the MAGIC_FIELDS_THAT_WORK_LIKE_-Ttext_-x part. The fields should give the offset, hex output in the middle like the above example, and ascii output on the right like the above example.

$ tshark -r $input_file -T fields -e frame.time -e MAGIC_FIELDS_THAT_WORK_LIKE_-Ttext_-x | awk -f remove_duplicates.awk | text2pcap -t "%b %d, %Y %H:%M:%S." -q - $output_file

Please note that the -t argument to text2pcap didn't actually work for me. I had to reformat the time a little before it worked. Also, I realize that the packets aren't 100% identical, since the layer 2 information is changed with different MAC's and TTL's. But I want to look only at layer 3 and above for my filtering.

Thanks for any help you can provide!

asked 12 Mar '13, 08:08

Lemurshark's gravatar image

Lemurshark
26569
accept rate: 0%

edited 12 Mar '13, 09:03


Does editcap with the -d option (along with the -D and -w options that set the frame window and time window for dups respectively) not meet your needs? Editcap should be installed along with Wireshark.

permanent link

answered 12 Mar '13, 08:13

grahamb's gravatar image

grahamb ♦
19.8k330206
accept rate: 22%

I believe that editcap with -d requires 100% duplicate packets, down to layer 2 even. What I'm capturing is (for instance) a packet coming into a node and the same packet getting forwarded on to another node. So the packet data and layer 3 and above is the same, but the MACs are different and the TTL is different, which I think threw off editcap earlier. I'm also often capturing the same packets on different nodes and merging those capture files together. Again in that case the layer 3 and above is the same, but layer 2 is different.

(12 Mar '13, 08:27) Lemurshark

Ok. I failed on reading comprehension of your question.

(12 Mar '13, 09:00) grahamb ♦

I updated the text of the original post to reflect the nature of the duplicate packets. Thanks for your suggestion

(12 Mar '13, 09:00) Lemurshark
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×832
×41
×33
×31
×6

question asked: 12 Mar '13, 08:08

question was seen: 3,556 times

last updated: 12 Mar '13, 09:03

p​o​w​e​r​e​d by O​S​Q​A