Dropped packets by Wireshark/Dumpcap/WinDump

Question

Hello, We are trying to record some iSCSI traffic using Wireshark, and was consistently seeing dropped packets. We are running with a 1G buffer size, and rotating 5 files with 1G file size as well. The data is being saved to a RAMDisk, and the TEMP environment variables have also been changed to use the Ramdisk.

Switched to using dumpcap straight, and with larger buffer sizes like wireshark. Same with winDump.

The machine is a 2.4GHz, 12 core, machine, and CPU usage when these numbers were recorded is about 14% max, including the dumppcap. The machine has about 72G of memory, 40 of which is being used for the Ramdisk, and about 20 is unused when these numbers were recorded.

I am positive we are doing something wrong here, or missing something really obvious. Can someone please point us in the right direction?

Here is the dumpcap output

C:\Program Files\Wireshark>dumpcap.exe -i 2 -i 3 -B 1024 -b files:5 -w F:\Trace\2.pcap
dumpcap: Ring buffer requested, but no maximum capture file size or duration were specified.
Capturing on 2 interfaces
File: F:\Trace\2_00001_20151207074008.pcap
Packets captured: 537560
Packets received/dropped on interface \Device\NPF_{42D93B77-D5B2-4FAD-8901-E7C54D63ECAD}: 421222/722498 (36.8%)
Packets received/dropped on interface \Device\NPF_{B0462921-3F8D-44A0-B828-7FDF0FD0AB9B}: 653570/556621 (54.0%)

F:\ is the RAMDisk.

The current output on the machines is about 1.5 Gbps on each interface being captured. We see the same behavior even when we try to use a single interface

C:\Program Files\Wireshark>dumpcap.exe -i 2 -B 1024 -b files:5 -w F:\Trace\2.pcap
dumpcap: Ring buffer requested, but no maximum capture file size or duration were specified.
Capturing on \Device\NPF_{42D93B77-D5B2-4FAD-8901-E7C54D63ECAD}
File: F:\Trace\2_00001_20151207074218.pcap
Packets captured: 419766
Packets received/dropped on interface \Device\NPF_{42D93B77-D5B2-4FAD-8901-E7C54D63ECAD}: 419766/587700 (41.7%)

With Windump, this is what we see:

F:\WinDump>WinDump.exe -n -i \Device\NPF_{42D93B77-D5B2-4FAD-8901-E7C54D63ECAD} -s0 -C 1024 -W 5 -B 2048576 -w capture.pcap -p -U WinDump.exe: listening on \Device\NPF_{42D93B77-D5B2-4FAD-8901-E7C54D63ECAD}

2980237 packets captured 4512223 packets received by filter 1449839 packets dropped by kernel

Answer 1

0

Please try to lower the capture frame size to figure out if the buffer is filling up too fast or if there is another problem. Start with 1024, 512, 256, 128, etc.

dumpcap -i 2 -B 1024 -b files:5 -w F:\Trace\2.pca p dumpcap -s 512

Regards
Kurt

answered 07 Dec '15, 00:10

Kurt Knochner ♦
24.8k●10●39●237
accept rate: 15%

We need the entire packet unfortunately, this is iSCSI, so we can have multiple PDUs in a single packet. With such a fast disk, and a big buffer, why is it that even tcpdump cannot keep up?

(07 Dec '15, 08:31) Manish Sharma

I understood @Kurt Knochner's recommendation not as a solution but as a diagnostic step, allowing you to better localize the root cause of the issue.

(07 Dec '15, 11:22) sindy

Not much of a difference at different packet sizes also. Its really bugging me now..Even tried changing the default socket buffer recieve size. Not sure what the issue is at this point. Any other thoughts?

(08 Dec '15, 08:17) Manish Sharma

Then it'a most certainly what @Jasper wrote. If the network card is not optimized for capturing, you might loose frames. Just one hint: If the network card (in worst case) creates an interrupt for every single frame, you would simply overwhelm the CPU. So, those special cards (as well as some regular cards) usually have a buffer and they will only create an interrupt if that internal buffer is about to get full. This will speed up the capture process. There are other things the optimize together with drivers.

Furthermore Windows and Winpcap are probably not the best platforms to capture in high speed networks. Please try Linux and compare the results.

(08 Dec '15, 13:27) Kurt Knochner ♦

Answer 2

My guess is that you're not using a high speed capture card with specialized capture drivers for this. The rest of the machine specs are impressive, but "normal" network cards are not fast enough to deal with high loads of packets (not even on 1GBit/s links, and certainly not on 10G or higher). I guess you need to look at FPGA based capture cards to do this, e.g. Napatech, Accolade Technology, Fiberblaze etc.