We have about 30 Windows VMs under vSphere 5.1. I am using esxtop to monitor the network and I see some odd behavior. Under the %DroppedPacketsReceived (%DRPRX) column you can see the problem. This column will show zeros for all VMs for a few seconds then all the VMs will show packet loss for a few seconds, then back to zero. I have a Win7 machine with Wireshark installed on this host, you can see it at the bottom of the screen shot. When I do a packet capture: - Will it show these dropped packets? - If so what would I be looking for?
asked 06 Nov '13, 10:50  HendersonD |
2 Answers:
I figured it out This thread about half way through talks about VDS Health Check sending out broadcast packets https://communities.vmware.com/message/2280450 I disabled the Health Check and the dropped packets vanished. The Virtual Distributed Switch Health Check just looks at this switch and sees if it is setup correctly. Having the health check turned on generates broadcast packets. Not sure why my VMs are dropping these packets but turning it off is the solution. answered 07 Nov '13, 06:10 HendersonD I accepted the answer, as it solves the problem. (07 Nov '13, 06:18) Kurt Knochner ♦ |
From the VMware KB article:
So, to answer your question:
The virtual machine will only be able to capture traffic directed to it's own machine, and therefor you won't see the traffic of the other virtual machines. Answer to you question: No, you won't see any packets of other virtual machines. BTW: You won't see the 'dropped' packets anyway, as they are, well ... dropped ;-)). You can change the operational mode of the vSwitch to 'promiscuous' mode. In that mode, the Win7 system (with Wireshark) will see 'everything' (depends on the configuration of port groups, vlans, etc.).
However, that will probably cause even more problems (due to overload)!! One possible solution to your problem is described in the VMware KB article: raise receive buffer sizes. HOWEVER: it remains unclear what causes dropped frames for so many machines. It could be a network burst, a DoS attack or an error in the design. To figure out what's going on, I suggest to mirror the physical network port(s) of the VMware host and capture that traffic. Maybe there is a traffic spike in the IO stats, that correlates to those dropped frames shown in esxtop. If there is nothing, the network traffic could be caused by another virtual machine. In that case you can either capture the traffic in each virtual machine (probably to many) or on the virtual switch after you placed it in promiscuous mode. WARNING: Don't do that without talking to your local VMware guru!! As I said, that might cause even more (severe) problems. Another option: Capture on the VMware host itself with tcpdump And finally: See a blog post of @Jasper
Regards answered 07 Nov '13, 02:25  Kurt Knochner ♦ edited 07 Nov '13, 02:33 Thanks for all the great suggestions I have seen the two VMWare KB articles. I will give them another shot but the first time I changed the Small RX buffer and RX Ring Size on one VM I did not see any changes in dropped packets. This is happening on every one of my 40+ server VMs and all of my View desktops so I would hate to think that these two parameters needed to be changed on all of them. Kurt - I actually made a VM with Win7 and Wireshark installed. If you look at my screen shot above it is the last VM listed. I ran the capture using this VM and did not see a smoking gun. Again, this could be due to my ignorance on not knowing exactly what to look for in a packet capture or the packets are dropped between the virtual switch and the guest OS driver and therefore doing the capture inside the guest OS will never capture these dropped packets. If I do the capture on the physical switch by running a port-mirror, what types of things am I looking for that would indicate packets are being dropped? (07 Nov '13, 05:34) HendersonD Apparently you found the solution yourself (see your answer). Just for the records:
As I said, you won't see any 'dropped' traffic in the virtual machine if the packets are dropped by the driver, so Wireshark will never know about those packets, hence no 'smoking gun' visible in Wireshark (07 Nov '13, 06:18) Kurt Knochner ♦ |
Not a direct answer to your questions but just a hint, can this be related to one of this:
http://kb.vmware.com/kb/1010071
http://kb.vmware.com/kb/2039495