This is our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

Hello, We are trying to record some iSCSI traffic using Wireshark, and was consistently seeing dropped packets. We are running with a 1G buffer size, and rotating 5 files with 1G file size as well. The data is being saved to a RAMDisk, and the TEMP environment variables have also been changed to use the Ramdisk.

Switched to using dumpcap straight, and with larger buffer sizes like wireshark. Same with winDump.

The machine is a 2.4GHz, 12 core, machine, and CPU usage when these numbers were recorded is about 14% max, including the dumppcap. The machine has about 72G of memory, 40 of which is being used for the Ramdisk, and about 20 is unused when these numbers were recorded.

I am positive we are doing something wrong here, or missing something really obvious. Can someone please point us in the right direction?

Here is the dumpcap output

C:\Program Files\Wireshark>dumpcap.exe -i 2 -i 3 -B 1024 -b files:5 -w F:\Trace\2.pcap
dumpcap: Ring buffer requested, but no maximum capture file size or duration were specified.
Capturing on 2 interfaces
File: F:\Trace\2_00001_20151207074008.pcap
Packets captured: 537560
Packets received/dropped on interface \Device\NPF_{42D93B77-D5B2-4FAD-8901-E7C54D63ECAD}: 421222/722498 (36.8%)
Packets received/dropped on interface \Device\NPF_{B0462921-3F8D-44A0-B828-7FDF0FD0AB9B}: 653570/556621 (54.0%)

F:\ is the RAMDisk.

The current output on the machines is about 1.5 Gbps on each interface being captured. We see the same behavior even when we try to use a single interface

C:\Program Files\Wireshark>dumpcap.exe -i 2 -B 1024 -b files:5 -w F:\Trace\2.pcap
dumpcap: Ring buffer requested, but no maximum capture file size or duration were specified.
Capturing on \Device\NPF_{42D93B77-D5B2-4FAD-8901-E7C54D63ECAD}
File: F:\Trace\2_00001_20151207074218.pcap
Packets captured: 419766
Packets received/dropped on interface \Device\NPF_{42D93B77-D5B2-4FAD-8901-E7C54D63ECAD}: 419766/587700 (41.7%)

With Windump, this is what we see:

F:\WinDump>WinDump.exe -n -i \Device\NPF_{42D93B77-D5B2-4FAD-8901-E7C54D63ECAD} -s0 -C 1024 -W 5 -B 2048576 -w capture.pcap -p -U
WinDump.exe: listening on \Device\NPF_{42D93B77-D5B2-4FAD-8901-E7C54D63ECAD}

2980237 packets captured
4512223 packets received by filter
1449839 packets dropped by kernel

asked 06 Dec '15, 23:51

Manish%20Sharma's gravatar image

Manish Sharma
1111
accept rate: 0%

edited 07 Dec '15, 00:11

grahamb's gravatar image

grahamb ♦
19.8k330206


Please try to lower the capture frame size to figure out if the buffer is filling up too fast or if there is another problem. Start with 1024, 512, 256, 128, etc.

dumpcap -i 2 -B 1024 -b files:5 -w F:\Trace\2.pca p dumpcap -s 512

Regards
Kurt

permanent link

answered 07 Dec '15, 00:10

Kurt%20Knochner's gravatar image

Kurt Knochner ♦
24.8k1039237
accept rate: 15%

We need the entire packet unfortunately, this is iSCSI, so we can have multiple PDUs in a single packet. With such a fast disk, and a big buffer, why is it that even tcpdump cannot keep up?

(07 Dec '15, 08:31) Manish Sharma

I understood @Kurt Knochner's recommendation not as a solution but as a diagnostic step, allowing you to better localize the root cause of the issue.

(07 Dec '15, 11:22) sindy

Not much of a difference at different packet sizes also. Its really bugging me now..Even tried changing the default socket buffer recieve size. Not sure what the issue is at this point. Any other thoughts?

(08 Dec '15, 08:17) Manish Sharma

Then it'a most certainly what @Jasper wrote. If the network card is not optimized for capturing, you might loose frames. Just one hint: If the network card (in worst case) creates an interrupt for every single frame, you would simply overwhelm the CPU. So, those special cards (as well as some regular cards) usually have a buffer and they will only create an interrupt if that internal buffer is about to get full. This will speed up the capture process. There are other things the optimize together with drivers.

Furthermore Windows and Winpcap are probably not the best platforms to capture in high speed networks. Please try Linux and compare the results.

(08 Dec '15, 13:27) Kurt Knochner ♦

My guess is that you're not using a high speed capture card with specialized capture drivers for this. The rest of the machine specs are impressive, but "normal" network cards are not fast enough to deal with high loads of packets (not even on 1GBit/s links, and certainly not on 10G or higher). I guess you need to look at FPGA based capture cards to do this, e.g. Napatech, Accolade Technology, Fiberblaze etc.

permanent link

answered 07 Dec '15, 00:11

Jasper's gravatar image

Jasper ♦♦
23.8k551284
accept rate: 18%

Its not a normal network card, its emulex 10G which is usually used specifically in SAN environments..Why is it that the capability to capture with such fast memory and CPU is a problem? I am having a hard time trying to understand where the bottleneck could be.

Is there something in the NIC card that can be changed?

(07 Dec '15, 08:33) Manish Sharma

You cod try NPCAP https://github.com/nmap/npcap I'm guessing that the single CPU core that's doing the capturing is at 100%

(07 Dec '15, 08:59) Anders ♦
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×1,620
×97
×89
×23
×2

question asked: 06 Dec '15, 23:51

question was seen: 3,866 times

last updated: 08 Dec '15, 13:27

p​o​w​e​r​e​d by O​S​Q​A