This is a static archive of our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

Wireshark leads to linux kernel crash when used on Vmware virtual NICs during nmap scans

0

I have a weird problem. I work with local penetration test environments on 2 different Opensuse 13.2 hosts. One laptop; one a desktop PC. Kernel version in both cases:3.16. VMware Workstation 11 is installed. Several Win XP and Win 7 guests can be started in the VMware environment. A host-only network is used for the VMware guests; the VMmware host only bridge got a virtual host NIC (vmnet3) assigned, too. (An additional VMware bridge to the physical device of the host is installed, too, but not actively used by the guests). In addition KVM with separate virtual networks/bridges and Linux guests is used on the very same host. The whole setup normally works totally stable and without any problems. Routing between the networks works flawlessly. The virtual bridges are in several test scenarios controlled by ebtables/iptables rules. However, the following problem is independent of whether netfilter is active or not.

To watch network packets passing the host NIC vmnet3 on their way to VMware guests I tried to use Wireshark or Tshark. This works - at least for normal traffic; vmnet3 is switched to promiscuous mode as expected. I can e.g. follow packets coming from KVM guests which are routed via the host to the VMware guest.

However, as soon as I start in addition a nmap scan on the Linux host directed against one of the VMware guests the whole Linux host system freezes. journalctl afterwards shows:

kernel: BUG: unable to handle kernel NULL pointer dereference at (null)

The host has to be restarted physically resetting/rebooting the machine. Which is a mess as all virtual machines get affected by the crash.

This bug is reproducible. It seems to be independent of different nmap scan types (I have not yet tested all.) It is independent of the VMmware guest type. It is independent of the hardware (laptop, desktop). It is independent of netfilter being active or not.

A similar effect does not occur if a scan is performed for KVM guests connected to KVm/qemu virtual networks/bridges. It only happens for VMware vmnet-host-devices. The combination of wireshark/tshark on virtual VMware host NICs plus the start of nmap against guests seems to lead to a kernel bug.

The bug does not occur if nmap scans are started from KVM linux guests against the vmware guests - i.e. not for routed packets. The bug occurs for nmap scans started on the host itself.

I should add that tcpdump in contrast to Wireshark/Tshark works flawlessly on the virtual VMware NIC (vmnet3). No problems for whatever nmap scan type executed.

So, this all leads me to the question whether there is a severe problem of Wireshark with virtual vmware NICs and certain ethernet packet created by nmap.

Any ideas?

Regards Ralph

asked 26 Oct '15, 04:57

moench's gravatar image

moench
6113
accept rate: 0%


One Answer:

0

So, this all leads me to the question whether there is a severe problem of Wireshark with virtual vmware NICs and certain ethernet packet created by nmap.

If you create a lot of new sessions very fast (nmap port scan), the dissection engine of Wireshark/tshark has to build internal hash table entries to keep track, so this sounds like a simple CPU overload situation. Why do you use Wireshark/tshark to do the capturing? Both are diagnosis/analysis tools, not capturing tools (although both can do that), which means in high load situations it's not a very good idea to do the capturing with Wireshark, as it's doing the analysis in parallel!

Please try to use dumpcap or tcpdump to capture the traffic and later use Wireshark/tshark to do the analysis!

Regards
Kurt

answered 26 Oct '15, 05:12

Kurt%20Knochner's gravatar image

Kurt Knochner ♦
24.8k1039237
accept rate: 15%

edited 26 Oct '15, 05:14

As already written I use tcpdump also - in this case I had/have to because of the failure of tshark/wireshark.

Nevertheless, I thought a bit about your answer. I accept the point being made about wireshark doing several things in parallel - but after some quick tests on the laptop I am not at all sure whether a CPU overload really explains the occurrence of the kernel bug.

Actually, the measured CPU load on even tiny time intervals is pretty small for just one VMware guest without any running special SW. Furthermore 8 CPU threads are available - and the VMware processes are assigned to only 4. But Ok, the low average CPU load on a 20 ms scale does not exclude a spike on smaller time scales.

But: Also simple nmap scans with only some created packets lead to the crash. Furthermore, you can parametrize the time scale for the packet creation of nmap. This does not seem to change anything. The crash seems to happen just a tiny bit later - but this is at best a marginal effect.

In addition I would expect something similar for virtual qemu/KVM host NICs if wireshark/tshark's interaction with virtual NIC devices really lead to CPU load problems. However, with KVM/qemu virtual NICs nothing happens - smooth error free capturing operation of wireshark/tshark.

So, I really don't know ...

(26 Oct '15, 05:45) moench

O.K. maybe there is a bug, but then I would wonder why a non-privileged process (Wireshark) is able to create a kernel crash (although in your question you mention just a freeze - 'whole Linux host system freezes'). Wireshark uses the same libpcap as tcpdump, so while there can be a bug in libpcap, triggered only by Wireshark, I think it's rather unlikely, but you never know...

Anyway: Can you please use dumpcap (actually beeing used by Wireshark/tshark to do the capturing) on that system. I would love to see if you experience a freeze as well.

(26 Oct '15, 06:35) Kurt Knochner ♦

Actually I "tested" too quickly on the laptop - and I must correct myself and the last comment. Sorry, for the misinformation ...

The "freeze" occurs also for pure KVM/qemu virtual host NICs (not bridged to a physical device). So, the problem occurs for both VMware AND KVM virtual NICs. I checked this also on the desktop machine

With the expression "kernel crash" I referred to the kernel bug message in the log journal of systemd:

kernel: BUG: unable to handle kernel NULL pointer dereference at (null)

which always appears as the last message directly before the "freeze".

Regarding dumpcap: I tested again - now for a KVM/qemu virtual host interface - result: freeze, with the same kernel related bug message in the systemd journal.

Regarding tcpdump: Works! No problems.

(26 Oct '15, 07:35) moench

O.K. then it could be a bug in dumpcap. What is your wireshark/dumpcap verson (-v will tell!)?

(26 Oct '15, 07:57) Kurt Knochner ♦

Thank you for the quick reaction !

dumpcap -v Dumpcap 1.12.8

...

Compiled (64-bit) with GLib 2.42.0, with libpcap, with libz 1.2.8, with POSIX capabilities (Linux), with libnl 3.

Running on Linux 3.16.7-24-desktop, with locale POSIX, with libpcap version 1.6.2, with libz 1.2.8. Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz

Built using gcc 4.8.3 20140627 [gcc-4_8-branch revision 212064].


tshark -v TShark 1.12.8 ..... Compiled (64-bit) with GLib 2.42.0, with libpcap, with libz 1.2.8, with POSIX capabilities (Linux), with libnl 3, with SMI 0.4.8, with c-ares 1.10.0, with Lua 5.2, without Python, with GnuTLS 3.2.18, with Gcrypt 1.6.1, with MIT Kerberos, with GeoIP.

Running on Linux 3.16.7-24-desktop, with locale POSIX, with libpcap version 1.6.2, with libz 1.2.8. Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz

Built using gcc 4.8.3 20140627 [gcc-4_8-branch revision 212064].

(26 Oct '15, 08:08) moench

1.12.8. Hm.. please open a bug report at https://bugs.wireshark.org . Add as much information as possible and a link to this question.

(26 Oct '15, 09:47) Kurt Knochner ♦
1

Or, even better, please open a bug report with the OpenSUSE maintainers, as a kernel crash is a kernel bug, not a bug of whatever program happens to trigger the kernel bug.

(26 Oct '15, 13:26) Guy Harris ♦♦

Also simple nmap scans with only some created packets lead to the crash.

what are the nmap parameters?

BTW: What's the output of the following command on the system that 'crashed'?

cat /proc/sys/net/core/bpf_jit_enable

If it's 1, try to disable BPF JIT

echo 0 > /proc/sys/net/core/bpf_jit_enable

(26 Oct '15, 14:34) Kurt Knochner ♦

Examples for simple TCP nmap parameters: -sT -T2 -p135 -v
or -sS -T2 -p135

What I wanted to do is to run "zombie" scans like nmap -sI Zombie_IP:port -Pn -p- --packet-trace -v Target_IP which actually works when I use tcpdump for packet capturing.

Regarding bfp_jit:

cat /proc/sys/net/core/bpf_jit_enable gives me 1, as soon as Wireshark/Tshark are started. Not before, not after.

However, setting it to "0" does not prevent the bug/freeze to occur.

I shall open a bug report tomorrow.

(26 Oct '15, 17:10) moench

What's the state of /proc/sys/net/core/bpf_jit_enable while you run tcpdump (and dumpcap)?

(26 Oct '15, 17:17) Kurt Knochner ♦
showing 5 of 10 show 5 more comments