tshark -T fields output same info and format as -T text -x

Hello,

I am trying to write a script that will remove duplicate packets (layer 3 and above) by comparing the data in two consecutive packets (or along a sliding window) and if they are identical, it will throw out the duplicates. The packets are not 100% duplicates since the MACs and TTLs are different, but from layer 3 and above they are identical. Therefore, editcap -d won't work, as it requires the MD5 hash to be the same.

The way I would like to do this is to output the data like this:

Mar 5, 2013 19:34:39.738281000 0000 00 15 f1 9c f8 00 00 01 fc 0d 68 c0 81 00 0c 90 ..........h..... 0010 08 10 46 a0 04 62 00 00 00 00 6e 11 05 c5 ac 10 ..E..c....~..... 0020 57 c6 ac 10 82 3e 13 c4 1a 0a 04 4f de 41 52 45 W....>.....O.ARE 0030 47 49 53 54 45 52 20 73 69 70 3a 66 61 70 2e 61 GISTER sip:data. ... 0460 36 2e 31 43 30 2e 36 32 3a 37 36 22 36 3b 6c 63 6.120.62:3626;fd 0470 3e 0d 0a 0d 0a >....

Then I can use a script to check the data for duplicates and use the timestamp at the beginning of the packet to read it back into a pcap file.

The command below summarizes what I have so far. What I need is the MAGIC_FIELDS_THAT_WORK_LIKE_-Ttext_-x part. The fields should give the offset, hex output in the middle like the above example, and ascii output on the right like the above example.

$ tshark -r $input_file -T fields -e frame.time -e MAGIC_FIELDS_THAT_WORK_LIKE_-Ttext_-x | awk -f remove_duplicates.awk | text2pcap -t "%b %d, %Y %H:%M:%S." -q - $output_file

Please note that the -t argument to text2pcap didn't actually work for me. I had to reformat the time a little before it worked. Also, I realize that the packets aren't 100% identical, since the layer 2 information is changed with different MAC's and TTL's. But I want to look only at layer 3 and above for my filtering.

Thanks for any help you can provide!