I am currently troubleshooting an issue with an IIS 8.5 / .NET 4.6.1 application running on Windows 2012R2. The host is a VMWare VM. I have a test script that simulates 5 users interacting with the application, and we are seeing requests / sec dropping 50%+ every 20-25 mins. This sluggishness persists for 5 or so minutes, and then things are good for another 20-25 mins. The cycle repeats. I have ruled our CPU, memory, DB calls. I have viewed more perfmon data than I'd care to admit. So I installed Wireshark to capture some traffic to troubleshoot further, and the problem spontaneously went away. I could not reproduce the issue. I have narrowed it down to this: When I start the VM anew, the problem described above is present and reproducible. However, if I start Wireshark (not capture, just start it), and test again, the problem is eliminated. The fix appears to be good, until a reboot. Then the problem returns. I have used procmon to try to figure out what Wireshark is doing to no avail. The PowerShell command: Get-NetAdapter | Format-List -Property "*" returns the same thing before and after starting Wireshark from a fresh boot. What could Wireshark be doing, possibly a Windows system call to configure the NIC in some way, that would explain this behavior? Thanks for any and all insight! -pc asked 02 Nov '16, 19:45  puppycrack showing 5 of 14 show 9 more comments |
This is a static archive of our old Q&A Site.
Please post any new questions and answers at ask.wireshark.org.
Have a look here about starting the NPF driver. Once you start Wireshark it looks at your network interfaces, which causes the driver to be started. This can also be done automatically at boot time. Try make that change and see if the problem is gone even before starting Wireshark (so having started the NPF driver only).
NPF is already set to auto-start. The key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NPF\Start is set to 0x2.
I have also verified the driver is started after reboot using: driverquery /FO list /v. State shows Running for the NPF module.
Very well, so after starting Wireshark (which starts looking at your network traffic right away) the problem seems to have gone. What about when you exit Wireshark again, does the problem come back?
Once I start Wireshark, the problem goes away. I can exit Wireshark, and things remain OK. The problem does, however, return after a reboot. I should also not that if I run "tshark tcp host 8.8.8.8", it returns with an error, as I wrongly assumed it used the same format as tcpdump. But this too has the same effect on the issue. If I run that command once after starting Windows, the problem is gone until a reboot.
Very well, this should be enough information. This is the point where the Windows guys jump in (and I'm not one of them)....
I can't think of an answer, except to point to VMWare in some way.
Wireshark doesn't do anything magical when it starts, it will spin up dumpcap (which will run some queries via WinPcap) to determine what interfaces are available (I don't think tshark does) at some point "phone home" to check for updates, tshark doesn't do that, and then if running the Qt version, queries each interface (frequently) for byte counts to create the spark lines.
You could try uninstalling WinPcap, restarting the VM and repeating the experiment, as it could be something in there "fixing" things.
The only thing which comes on my mind (and I'm not a Windows guy either) is that Wireshark recently (last couple of months) actually starts capturing on the background in order to be able to draw the graph of traffic volume over time in the interface list. I'm not sure whether tshark captures for a while even if you give it wrong parameters.
The important point is that starting a capture switches the network interface driver to promiscuous mode (the question is whether this is the case for VMware virtual NIC drivers as well), which may affect network stack operation as it can learn some MAC addresses which come in as source ones.
If you want to analyze the behaviour without affecting it, you have to use some other machine for capturing.
The whole story sounds to me like an Layer2 (ARP-cache) problem. And we have seen some problems here that sounds more or less the same. But it seems now, that @sindy has given the great hint. The traffic graph.
So you should disable the promiscous mode on all interfaces, this even survives a restart. And look if you can see your problem again.
I wrote a loop that runs "Get-NetAdapter | Format-List -Property PromiscuousMode" over and over. When I run the tshark command, I can see the NIC go into Promiscuous Mode for a split second. However, when I add the "-p" option to tshark (which instructs it to NOT put the NIC into promiscuous mode), the problem is still resolved. Running the same loop shows the NIC not going into promiscuous mode when run with the -p switch. I checked this a few times to make sure it wasn't a timing thing.
In that case, you may try to uninstall WinPcap alone, leaving Wireshark in place, and try to run Wireshark to see whether it still helps or not (it should not).
So I installed Wireshark, sans WinPcap, and found that starting either wireshark or tshark no longer resolves the issue. So WinPcap is responsible for whatever is going on. I guess I'll head over there to see if anyone can shed some light on this. Thanks everyone!
I would try to gather even more info by using NPcap instead of WinPcap. The difference between the two is where the capturing filter is inserted: WinPcap uses NDIS 5 and NPcap uses NDIS 6. In the best case, use of NPcap will allow you to see the actual issue happening as it will not fix it.
WinPcap is basically on life support, it works well as it ever did in it's current version, but no changes are planned. Npcap is under active development.
Both WinPcap and npcap insert themselves in the driver stack handling protocol data. Presumably this affects something in your misbehaving system.