Hello, We are trying to record some iSCSI traffic using Wireshark, and was consistently seeing dropped packets. We are running with a 1G buffer size, and rotating 5 files with 1G file size as well. The data is being saved to a RAMDisk, and the TEMP environment variables have also been changed to use the Ramdisk. Switched to using dumpcap straight, and with larger buffer sizes like wireshark. Same with winDump. The machine is a 2.4GHz, 12 core, machine, and CPU usage when these numbers were recorded is about 14% max, including the dumppcap. The machine has about 72G of memory, 40 of which is being used for the Ramdisk, and about 20 is unused when these numbers were recorded. I am positive we are doing something wrong here, or missing something really obvious. Can someone please point us in the right direction? Here is the dumpcap output
F:\ is the RAMDisk. The current output on the machines is about 1.5 Gbps on each interface being captured. We see the same behavior even when we try to use a single interface
With Windump, this is what we see:
|
Please try to lower the capture frame size to figure out if the buffer is filling up too fast or if there is another problem. Start with 1024, 512, 256, 128, etc.
Regards We need the entire packet unfortunately, this is iSCSI, so we can have multiple PDUs in a single packet. With such a fast disk, and a big buffer, why is it that even tcpdump cannot keep up?
(07 Dec '15, 08:31)
Manish Sharma
I understood @Kurt Knochner's recommendation not as a solution but as a diagnostic step, allowing you to better localize the root cause of the issue.
(07 Dec '15, 11:22)
sindy
Not much of a difference at different packet sizes also. Its really bugging me now..Even tried changing the default socket buffer recieve size. Not sure what the issue is at this point. Any other thoughts?
(08 Dec '15, 08:17)
Manish Sharma
Then it'a most certainly what @Jasper wrote. If the network card is not optimized for capturing, you might loose frames. Just one hint: If the network card (in worst case) creates an interrupt for every single frame, you would simply overwhelm the CPU. So, those special cards (as well as some regular cards) usually have a buffer and they will only create an interrupt if that internal buffer is about to get full. This will speed up the capture process. There are other things the optimize together with drivers. Furthermore Windows and Winpcap are probably not the best platforms to capture in high speed networks. Please try Linux and compare the results.
(08 Dec '15, 13:27)
Kurt Knochner ♦
|
My guess is that you're not using a high speed capture card with specialized capture drivers for this. The rest of the machine specs are impressive, but "normal" network cards are not fast enough to deal with high loads of packets (not even on 1GBit/s links, and certainly not on 10G or higher). I guess you need to look at FPGA based capture cards to do this, e.g. Napatech, Accolade Technology, Fiberblaze etc. Its not a normal network card, its emulex 10G which is usually used specifically in SAN environments..Why is it that the capability to capture with such fast memory and CPU is a problem? I am having a hard time trying to understand where the bottleneck could be. Is there something in the NIC card that can be changed?
(07 Dec '15, 08:33)
Manish Sharma
You cod try NPCAP https://github.com/nmap/npcap I'm guessing that the single CPU core that's doing the capturing is at 100%
(07 Dec '15, 08:59)
Anders ♦
|