This is a very odd problem we have run into... We have configured a many-to-one port mirror on a switch to forward all traffic from the mirrored servers to a Linux monitor server. We tested the many-to-one port mirror with laptops pinging each other and used another laptop running Wireshark 1.0.2 to capture the traffic, proving that everything is working. When we added the Linux monitor server to the mix the monitor shows that it is only getting the traffic being transmitted from the mirrored server ports and not getting the traffic being received on those ports. We moved the connection from the Linux monitor server to the laptop with Wireshark 1.0.2 and ran a new capture which showed both the transmit and receive traffic, in this case pings to the servers on the mirrored ports and their replies to the PC on the LAN sending the pings. When a SysAdmin put a laptop running Wireshark 1.8.5 on the connection and ran a new capture of the still running pings his capture only showed the transmit traffic from the mirrored servers, the ping replies, and did not show the pings the servers were receiving. We immediately swapped back to the Wireshark 1.0.2 laptop and confirmed that it was capturing both the pings and the replies, the transmit and received traffic from the mirrored servers. A tcpdump from a Linux laptop on that same connection also captured the pings and replies, transmited and received traffic from the servers. We compared the capture settings we were using for Wireshark 1.0.2 and Wireshark 1.8.5 and they were the same. Both were configured to capture packets in promiscuous mode with a 5 megabyte buffer. Both laptops were auto-negotiating at full Gig, though the Wireshark 1.0.2 laptop is running Windows XP SP3 and the Wireshark 1.8.5 laptop is running Windows 7. In both cases we did not have any capture filters applied. We are stumped on this one. The Wireshark 1.0.2 captures confirm that all the traffic, transmited and reveived, is being mirred from the servers, but Wireshark 1.8.5 and the Linux monitor server are only capturing the traffic being transmitted by the mirrored servers. Has anyone else ever seen or heard of this issue? Many Thanks! Jandrys asked 01 Mar '13, 10:30  Jandrysik edited 01 Mar '13, 15:13  Guy Harris ♦♦ showing 5 of 22 show 17 more comments |
This is a static archive of our old Q&A Site.
Please post any new questions and answers at ask.wireshark.org.
Hi,
I'm a bit confused when and what did or did not work. Could you please put together two cases where you describe which system was connected to which switch port (just symbolic), who was talking to whom (symbolic names) and what you did see (or not).
Thanks
Kurt
Kurt,
Here are four cases, two successes and two failures.
Situation:
PC A is pinging Server B and Server C. Server B and Server C are both on the same network and the same switch. The switch has a 'many-to-one' port mirror configured to forward all traffic sent and received (Tx/Rx) by Server B and Server C to a monitor port.
Case 1 - SUCCESS:
A 32-bit Windows XP laptop running Wireshark 1.0.2 is connected to the monitor port on the switch. The Wireshark capture has promiscuous mode enabled and is using a 5 megabyte buffer but no capture filter. The Windows firewall is disabled on the laptop.
Wireshark 1.0.2 is able to capture the traffic transmitted and received by Server B and C.
When the IP address of PC A is used as a viewing filter Wireshark displays the pings from PC A to Servers B and C, and the ping replies from Servers B and C to PC A.
Case 2 - FAILURE:
A 64-bit Windows 7 laptop running Wireshark 1.8.5 is connected to the monitor port on the switch. The Wireshark capture has promiscuous mode enabled and is using a 5 megabyte buffer but no capture filter. The Windows firewall is disabled on the laptop.
Wireshark 1.8.5 is ONLY able to capture the traffic transmitted by Servers B and C.
When the IP address of PC A is used as a viewing filter Wireshark ONLY displays the ping replies from Servers B and C to PC A. Wireshark does NOT display the original pings from PC A to Servers B and C.
Case 3 - SUCCESS:
A Linux laptop is connected to the monitor port on the switch.
A tcpdump command on the laptop's NIC is able to capture the traffic transmitted and received by Server B and C.
When the tcpdump is run using the IP address of PC A as a filter the capture displays the pings from PC A to Servers B and C, and the ping replies from Servers B and C to PC A.
Case 4 - FAILURE:
The Linux monitor server is connected to the monitor port on the switch.
A tcpdump on the Linux monitor server's NIC is ONLY able to capture the traffic transmitted by Servers B and C.
When the tcpdump is run using the IP address of PC A as a filter the capture ONLY displays the ping replies from Servers B and C to PC A. The capture does NOT display the original pings from PC A to Servers B and C.
Our Network Engineers and System Administrators are all scratching their heads on this one.
Please let me know if you have any questions, comments or recommendations regarding this issue. Thanks! Jandrys
How is this Linux monitor server setup? Is it a dedicated platform? Standalone, like the Linux laptop, or part of a regular server, of a virtual machine?
The Linux monitor server is a dedicated standalone Red Hat 6 server.
Thanks, Jandrysik
do your captures contain VLAN tags?
Landi, No, the two mirrored server ports and the monitor port are all untagged and on the same VLAN. Thanks, Jandrysik
Landi, I pulled up one of the successful captures, checked the IP flags in the packets and did not see any VLAN tags. Thanks, Jandrysik
Our SysAdmin team suspects this is an issue with 64-bit vs 32-bit OS versions. They are doing some more testing and are going to reload the Linux monitor server with 32-bit OS.
They are also going to re-install Wireshark 1.8.5 on 32-bit Windows laptop to see if it will correctly capture both the transmitted and received traffic from the mirrored ports.
Thanks, Jandrysik
I don't think this problem is related to Wireshark and/or 32/64 Bit, as Case 3 works on a Linux Laptop and Case 4 does not work on another Linux system, both using tcpdump and not Wireshark.
Did you check if the capturing interface negotiated the right speed and duplex with the switch (Gigabit / Full) in Case 2 and 4 (Failure)?
Kurt, That was one of the first things we checked during the captures. All of the interfaces involved are auto-negotiating at full Gigabit, including on the failed attempts.
Please keep in mind that the failed captures are perfect examples of a Tx / transmit only port mirror capture. No malformed or partial packets. They are nice, clean captures of only the transmit traffic.
I would have expected to see malformed packets if there were auto-negotiation problems between the switch monitor port and the laptops / server.
Thanks for the suggestion though! Jandrysik
well, just an idea ;-)
Unfortunately this makes no sense at all as there is no common part in the four cases that could cause/explain the problem !??!
What are the Linux versions on the Laptop and on the Monitoring server? What types of interfaces (brand/modell) did you use on Linux (dmesg | grep eth0)?
BTW: Did you check if a local firewall blocked the packets (as stupid as this may sound).
http://ask.wireshark.org/questions/10377/when-i-capture-ping-test-in-wireshark-i-only-see-the-reply-and-no-request-why
Server is a Dell R720 with the the Broadcom 5720 quad port card. Server is running RedHat EL 6.4 x64bit.
SysAdmin test laptop is a Dell Latitude E5500 with a Broadcomm 5756. 4 different OS's
And yes, the local firewalls are disabled on the laptops and the Linux monitor server.
Thanks again to everyone for the help and suggestions regarding this problem. Jandrysik
So, it depends on the OS if tcpdump works, on the same hardware? It does not work with RHEL and it works with Fedora? Very strange!!
Do you have any IP configuration on the capturing interfaces in either case? If so, can you please try to disable that IP configuration and run the interface just as a capturing interfaces (cli: ifconfig eth0 0.0.0.0).
Kurt, Yep, this is a odd one. Different OS running on the exact same hardware and tcpdump either works or fails depending on the OS.
Nope, we do not have an IP configurated on the capturing interfaces on the laptops or the Linux monitor server.
Our SysAdmin is still waiting to hear back from Red Hat regarding this.
Thanks again! Jandrysik
can you try CentOS 6.3 instead of RHEL 6.3? Just curios….
Kurt, The SysAdmins had a copy of CentOS 6.2 that they tried. It failed as well… Thanks, Jandrysik
Unbelievable…. Please post the solution to this problem if you ever find it! Thanks Kurt