This is a static archive of our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

Performance Windows vs Linux capture only

0

At work I've been trying to setup a machine with the sole purpose of capturing traffic for later analysis.

Linux seemed like the best option since it required very little resources. I've tried tshark, dumpcap and tcpdump. With all 3 command line tools it displays packet drops as high as 20% (I'm assuming because the hardware can't handle the amount of traffic). So I tried with Windows and the number of dropped packets was 0...

Can anyone relate? Could this be the difference in libpcap and winpcap?

Test traffic (troughput and such) was the same.

Software used: Linux; Debian 7; libpcap v1.3.0-1; tshark v1.8.2-5; tcpdump v4.3.0-1 Windows; Windows XP SP3; winpcap v4.1.3; wireshark v1.10.2

asked 16 Oct '13, 03:49

psmul's gravatar image

psmul
11125
accept rate: 50%

1

How many packet/s?(or Mbs) It's possible Winpcap does not report packet loss accurately do you have any way of verifying 0 packet drops and no drops on the interface?

(16 Oct '13, 04:17) Anders ♦

I've tried to relate the packet drops to the number of "TCP segment not seen" (the 'these are normal during capture start' warnings in the expert window) and as suspected, Windows shows only 8 or 10 while linux capture files show more then 1000

(16 Oct '13, 04:44) psmul

Try tshark with -B 2 and -s 2000 on Linux to see if it makes a difference.

(16 Oct '13, 05:47) Anders ♦

Doesnt that mean I will capture less data per packet (so I might lose relevant data?)

ps. i've tried the -B option from 2MB - 25MB but no difference

(16 Oct '13, 05:59) psmul

Do you see any packet over 2000 bytes?

(16 Oct '13, 06:26) Anders ♦

So far so good. I'm now running a test that will last for 30+ minutes. In the mean time, what affect will this have when analysing? Will it (most likely) cut data short (which means I can't read out the data) but still be able to see protocol, ports etc?

(16 Oct '13, 06:40) psmul
1

Presumably if MTU ~1500 you will not miss anything unless there is jumboframes. If I undestod/remember correctly there is a problem with libpcap reserving buffer space of 65535 if no snaplength is defined making inefficient use of the buffer(s). I'm not sure which versions of libpcap has this problem(if it is a problem).

(16 Oct '13, 06:47) Anders ♦

Tried a snaplenght of 30.000 and still dropped some frames, Btw, just compiled latest version of libpcap and tcpdump and still same problem. Anyway, I'll let you know in a minute

(16 Oct '13, 06:55) psmul

Too bad. 4,041,089 packets captured, 73,422 packets dropped :S

(16 Oct '13, 07:15) psmul

Are you using capture filters? What HW do you use on Linux.

(16 Oct '13, 08:59) Anders ♦

No capture filters, fujitsu siemens esprimo D9510 laptop (test only)

(16 Oct '13, 11:53) psmul

Same hardware for Linux and Windows?

(16 Oct '13, 12:11) Kurt Knochner ♦
1

Tried a snaplenght of 30.000 and still dropped some frames

If you mean 3*10^4 (i.e., the . is a thousands separator), that snapshot length is much higher than you need.

If this is Ethernet, and there are no jumbo frames, try a snapshot length of 1514.

(The problem to which Anders is referring is that the memory-mapped capture mechanism uses fixed-length slots for packets, so the slot size has to be the maximum packet size if the entire packet is to be captured. libpcap originally used the snapshot length for this, but the default snapshot length used by Wireshark/TShark/dumpcap is 65535, chosen to be "large enough", which is larger than the actual maximum snapshot size. Newer versions of libpcap will, on Ethernet, try to determine a smaller size and, if they can determine a smaller size, will use it.)

(16 Oct '13, 12:25) Guy Harris ♦♦

@Kurt: yes @Guy: I don't think I get it. Why would this be a problem? Concerning testing it with actually 1514, I will have to try tomorrow. I'll keep you posted

(16 Oct '13, 12:32) psmul

Just for the fun of things, do you guys ever capture traffic using command line only tools? And do you abort them manually (so you can actually see any dropped packets when it closes down).

For I didn't even notice these errors while testing because my command line (was supposed to) start on bootup. When troubleshooting my Linux configuration I noticed it actually dropped a lot of traffic. In other words, it would have gone undetected (except for the lost tcp segements in expert info) if it wasn't for manually start and kill the command line tools.

(16 Oct '13, 12:40) psmul
1

Why would this be a problem?

It would be a problem because the size of the memory-mapped buffer is limited (make it too big and the kernel will refuse to let you allocate a chunk of wired-down kernel memory that large), and the bigger the per-packet slots are the fewer slots there are for packets are the buffer.

(16 Oct '13, 12:43) Guy Harris ♦♦
1

If you are using the same HW for Linux and Windows do you run Linix in a VM or do you have dual boot? I guess running in a VM may impact performance.

(16 Oct '13, 13:10) Anders ♦

so, capturing with -s0 (65k) is a bad idea, as there are fewer slots for packets in the kernel buffer?

(16 Oct '13, 13:35) Kurt Knochner ♦
2

so, capturing with -s0 (65k) is a bad idea, as there are fewer slots for packets in the kernel buffer?

When capturing with the TPACKET_V1 and TPACKET_V2 memory-mapped capture mechanisms (which are the only memory-mapped capture mechanisms supported by current libpcap - TPACKET_V3 isn't currently supported - and which are used by libpcap if available), the larger the maximum packet size libpcap uses, the fewer packet slots there are.

With newer versions of libpcap, even with -s0 (65K snapshot length) will attempt to pick a "better" maximum packet size, i.e. the smallest needed to ensure that no packets are cut short. If there's segmentation offloading (so that a "packet" delivered to the adapter could be bigger than the maximum Ethernet packet size), or if the adapter isn't an Ethernet adapter, libpcap will fall back on the snapshot length as the maximum packet size, otherwise it'll use the MTU + 14 bytes for the Ethernet header.

So -s0 may mean "fewer slots", but it may also mean "you get all the bytes of the packet". Manually choosing an "appropriate" snapshot length would work better, but that's link-layer header type dependent, and some such headers (e.g., radiotap headers) are variable-length, so it's tricky.

(16 Oct '13, 14:00) Guy Harris ♦♦

Interesting. I've never thought about the relationship between capture size and buffer space, as I did not know about those internals of libpcap. Thanks for pointing me to that 'issue'.

(16 Oct '13, 16:11) Kurt Knochner ♦

@Anders: No, not running in a VM. I just reinstall now and then.

(16 Oct '13, 22:31) psmul
1

@psmul I usually capture on the command line using dumpcap when it is a capture that runs for more than a few minutes, and abort the capture by CTRL-C.

Since I use the PCAPng capture file format I can see the dropped frame count per file, because PCAPng does keep the number of dropped frames that dumpcap would have displayed. The drop count may not be completely accurate because there are multiple opportunities for packets to be dropped before making it into the capture file, and not all of them are noticed by dumpcap.

(16 Oct '13, 22:31) Jasper ♦♦

Using Ubuntu and snaplength of 1514, 1.179.216 packets captures, 0 drops! Looking good. Just going to reinstalling Debian, be back in an hour with test results

(17 Oct '13, 00:42) psmul

Debian, new compiled libpcap and tcpdump, 1.831.616 packets captured, 3806 drops. I'll try again with the default repository versions. You figure the problem could be in the os ... ?!

(17 Oct '13, 02:09) psmul
1

Can you please just boot your Laptop with Kali Linux, instead of installing Debian? Then capture there with tcpdump and/or dumpcap. Do you get the same results? If yes: strange. If no: Debian problem.

(17 Oct '13, 02:20) Kurt Knochner ♦

Just did, 1.257.325 packets captured, 277 packets dropped. I'd say that could be due to hardware and high throughput. All in all, not to bad. I'll try again on my raspberry (this time with the Arch version instead of Raspbian) see what it does this time.

Edit: Just ran it again, 1.781.782 packets captures, 0 drops.

Could this be driver issues? Any other plausible explanation?

(17 Oct '13, 04:42) psmul

Just did,

with Kali, Debian or Windows?

(17 Oct '13, 04:44) Kurt Knochner ♦

Sorry, with Kali

(17 Oct '13, 04:50) psmul

O.K. so you do have more drops with Debian than with Kali? If so, it might be related to difference in the libpcap version or the kernel (driver, etc.)

(17 Oct '13, 05:40) Kurt Knochner ♦

Raspberry still not working correctly, but chances are high its the hardware that can't handle the load. Anyway I'm marking this as solved and will stick to capturing on other distro's

Thanks for the assistance.

(17 Oct '13, 07:15) psmul
1

If some one is interested a co-worker just submitted a pull request to libpcap implementing TPACKET_v3 support https://github.com/the-tcpdump-group/libpcap/pull/325

(17 Oct '13, 07:23) Anders ♦

@Jasper:

Since I use the PCAPng capture file format I can see the dropped frame count per file, because PCAPng does keep the number of dropped frames that dumpcap would have displayed.

How does this work? I know the command line drops packets (it shows me when I kill command line) but when opening in Wireshark it does not show in the statusbar.

(17 Oct '13, 07:50) psmul
1

That's because Wireshark records the dropped packets per interface (which makes sense). The drop count in the status bar is only updated right after dumpcap is stopped.

To see the interface statistics after opening a file, go to Statistics -> Sumary and take a look at the interface table in the middle of the dialog window.

(17 Oct '13, 08:08) Jasper ♦♦

used in the same test.

well, it was not the same test. If you want to compare results, you need a packet generator that generates the same data every time you test. Then you capture 10 times (or more) with Linux and 10 times with windows. You need more than a few tests with identical test parameters, to filter out CPU usage peaks or IO peaks that might have an influence on the test result. Then compare the content of the capture files with what you expected to get from the packet generator.

With that test method you would get far better and more realistic results ;-)) I'm not saying that windows might not perform better on a certain platform, but without a proper test environment (see differences of Debian and Kali!) there is a good chance to learn the wrong things from an experiment ;-))

(17 Oct '13, 09:30) Kurt Knochner ♦

Very true,

Just couldn't believe I'm part of only a select few who was bothered by this. I was hoping, that more network admins (and sorts) were using cli linux for packet capturing and already knew why this happened or had a solution for this problem.

It didn't occur to me to spend that much time testing since the problem between Windows and Linux seemed so obvious. The more I was testing, the more blurred the lines became :)

I'll have to wait to get back to work tomorrow, but I'm gonna give that pf_ring a try. Maybe the problem is instantly solved... In the mean time hoping someone could verify this idea/solution.

(17 Oct '13, 11:14) psmul

Would be logical to leave/display it at the statusbar.

It probably would be. Please file an enhancement request on the Wireshark bugzilla.

(17 Oct '13, 11:41) Guy Harris ♦♦

Found this interesting article: Improving Passive Packet Capture: Beyond Device Polling

It might be interesting to see results with newer versions of {Windows,Linux,FreeBSD,WinPcap,libpcap}.

(17 Oct '13, 11:43) Guy Harris ♦♦

Would there be any rationale why WinPcap should be faster than libpcap?

(17 Oct '13, 12:42) Kurt Knochner ♦

Would there be any rationale why WinPcap should be faster than libpcap?

Given that a lot of the performance issues are with the OS and its packet capture mechanism, you can't say anything is faster than or slower than libpcap without specifying the platform on which you're running libpcap; there's no generic reason why WinPcap would be faster than libpcap on all platforms other than "none of the UN*Xes on which libpcap runs have a capture mechanism that works as well as NDIS+the WinPcap driver".

The performance would also depend on the OS version and the libpcap/WinPcap version.

(17 Oct '13, 12:58) Guy Harris ♦♦

there's no generic reason why WinPcap would be faster than libpcap on all platforms

I tend to agree with that, although I don't know the internals of WinPcap or libpcap good enough for my own (internal) final résumé.

I've found this:

http://www.winpcap.org/docs/docs_412/html/group__NPF.html

Cite: 'When the kernel-level traffic logging feature of NPF is enabled, the capture driver addresses the file system directly, hence the path covered by the packets is the one of the red dotted arrow: only two buffers and a single copy are necessary, the number of system call is drastically reduced, therefore the performance is considerably better.'

That sounds like a (possible) advantage. Is there anything similar in libpcap?

The function pcap_live_dump() is not used in dumpcap, so there is no way to use/test that right now. However, there is sample code for a tool called kdump (at the end of the page).

Would be interesting to see the difference of pcap_dump() in dumpcap and pcap_live_dump() in kdump. And that compared to libpcap on the systems you mentioned.

(17 Oct '13, 14:35) Kurt Knochner ♦
showing 5 of 40 show 35 more comments

2 Answers:

0

@Jasper:

Thanks for clearing that up. Would be logical to leave/display it at the statusbar. Anyway, without going off topic.

Has anyone ever experimented with pf_ring before ... ? Found this interesting article: Improving Passive Packet Capture: Beyond Device Polling

I was wondering if this could boost performance. It actually describes everything I experienced.

The experience learnt from this experiment is:
• Out of the box, Windows and winpcap [winpcap], the port of libpcap to Win32, perform much better than other popular Unix-like OS.
• Linux, a very popular OS used for running network appliances, performs very poorly with respect to other OSs used in the same test.

answered 17 Oct '13, 09:03

psmul's gravatar image

psmul
11125
accept rate: 50%

Tested PF_RING several times on Debian (in my own little test setup, so nothing fancy) and worked like a charm. All captures went 1.000.000 + packets captured and not a single dropped packet. I'd say this solves my problem.

For future reference: PF_RING by ntop.org

(17 Oct '13, 23:52) psmul

If you have the time to try the patch for TPACKET_V3 to libpcap as well and perhaps measure CPU load for the two cases that would be helpful too.

(18 Oct '13, 03:41) Anders ♦

0

Problems arose on Raspbian and Debian. After better results on Ubuntu and near perfect results in Kali, I figure that's the problem.

Thanks for all the assistance.

answered 17 Oct '13, 07:16

psmul's gravatar image

psmul
11125
accept rate: 50%