Removing duplicate packets

Question

I have found that when mirroring packets on at least some Cisco platforms, in some cases the ehthernet MAC address is that of the sending/receiving device, and in others it's the MAC address of the switch -- this is true for the same TCP session and creates duplicates. When this happens, editcap -d does not remove the duplicates. Is there a way to cause editcap to ignore the ehthernet portion when de-duplicating? Thanks.

Answer 1

4

The MAC address of the switch should not appear in a layer 3 packet since the switch is a layer 2 device - unless you are running a layer 3 switch. And if the MAC address is different it is not a duplicate (and editcap is right in not removing them), unless you want to ignore layer 2.

Usually you get two frames identical except for layer 2 when using VLANs, so you're seeing the frame in one VLAN and then also in another after it was routed. Have you checked if the frames are in different VLANs? Keep in mind that, depending on your capture card, you might not be able to record the VLAN tag even though it's in the original frame coming in - that way you see the same frame with different MAC addresses without VLAN tag because the card didn't record it.

answered 19 Jul '11, 16:46

Jasper ♦♦
23.8k●5●51●284
accept rate: 18%

edited 19 Jul '11, 16:46

In most cases, I'm not concerned with the ethernet layer -- I would just like to have a "clean" TCP/IP layer trace to go over with the application folks. I hate having to explain that the duplicates are normal and to be expected.

(20 Jul '11, 07:48) jowimi

1

Ok, I see. Did you check if your duplicates are due to seeing a routed frame twice? You can detect this by looking at VLAN tags or the TTL - if the same packet has 2 different TTLs (usually +/-1) it got routed in the meantime. BTW, I forgot to mention another reason for missing VLAN tags: the SPAN port strips them, too, if not told otherwise; use the "encapsulation dot1q" command (if you're using cisco devices to include them, otherwise check your manuals).

If you can find different VLANs or TTLs in the duplicate and original you can easily filter one group away and save the rest, dupe free.

(20 Jul '11, 07:53) Jasper ♦♦

I editcap -d'd a trace first -- that got rid of the clearly duplicate packets. Then I applied a display filter ip.ttl == nn or ip.ttl == nn to include the TTL for each side of the non-duoplicated packets. Worked great!! Thanks for the advice.

(20 Jul '11, 09:10) jowimi

Good to see it worked itself out :-) You might want to accept the answer to get it out of the ton of "unanswered" questions.

(20 Jul '11, 09:13) Jasper ♦♦

This thread was JUST what I was looking for. I'm seeing "dupes" and, like jowimi, "I hate having to explain that the duplicates are normal and to be expected." And my dupes DO seem to be caused by VLAN routing, as their TTL's are off by one.

However, my cranial density prevents me from grasping the solution :-). jowimi says "I applied a display filter ip.ttl == nn or ip.ttl == nn", and it got rid of the dupes. In my case, I see TTL's of my dupes as 61 & 62, and then sometimes they are 63 & 64, and i could maybe find other pairs if I kept looking. What do i set "nn" to, and how does it work?

Thx guys - GREAT stuff!

feenyman99

(04 Aug '12, 07:30) feenyman99

You can use display filter:
ip.ttl==61 || ip.ttl=62 || ip.ttl==63 || ip.ttl==63
Or you can use a display filter like:
ip.ttl>60 && ip.ttl<70

(04 Aug ‘12, 23:42) joke

The best way for you to remove the dups is to filter on vlan-id or mac-addresses and then either save the resulting packets to a new file or choose to ignore one side of the conversation with “Edit -> Ignore all displayed packets”

(05 Aug ‘12, 22:55) SYN-bit ♦♦

showing 5 of 7 show 2 more comments

Answer 2

Jasper's tips work great when you have a tracefile that has traffic of two sides of a router. A good way to prevent these duplicates to turn up in your capture file is to narrow down the configuration of your monitor session.

In case there are multiple vlans on a switchport it might be better to span the vlan instead of the physical port. Beware to capture only RX packets and not both TX and RX, as each packet that enters the switch on a particular vlan usually leaves the switch on that vlan too (even if it is just to the routing engine). It's always a good idea to run a (few) test captures until you are sure you see all traffic of interest and only traffic of interest :-)