This is a static archive of our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

Tshark stops working at random times

Hi All,

First off, i am running Kali 2. on a Raspberry Pi 2b with a TP-Link TL-WN722N USB WiFi Dongle.

Running tshark -v displays version 1.12.6 and running on Linux 3.18.16v7 with locale, with libpcap version 1.6.2 with libz 1.2.8.

I hope that helps...

So on to the problem.

I ( myself and a buddy ) are trying to use the RPI to capture the number of wireless devices in a street for a study we are working on. Before anyone asks... yes i do have permission :-)

The RPI works great, it feeds the data back to a mini web portal we had written that displays the data graphically and on first boot up everything works great. ** disclaimer... We had quite a bit of help setting this up and unfortunately the guy helping has since moved on and no longer contactable.. hence the question here and also why parts of it may come across as being a noob... which we are to using Tshark / Wireshark.

After random periods the RPI stops sending data, we can log into it, ping the web server plus lots of other web related sites, the memory card is only around 30 % full... ( 16Gb in total ) and a memory check suggests there is plenty available... in essence everything seems fine.

If we stop and start Tshark service it makes no difference, we still do not receive any data. The only way to resolve it is to reboot the RPI.

Although i can not prove it i think the issue is related to the amount of data the RPI is receiving / sending. Reason being is throughout a night time the RPI lasts hours.. however on a normal day around lunch time when the street is at its busiest the data transmit stops and we have to reboot.

The command line we are running is as follows

/usr/bin/nohup /usr/bin/tshark -l -I -i wlan1 -T fields -e frame.time_epoch -e wlan.sa -e radiotap.dbm_antsignal -e wlan_mgt.ssid -E occurrence=f type mgt subtype probe-req | /usr/bin/php /opt/dronecapture/tsharkparser.php&

The tsharkparser.php file the command line refers to is as follows

<?php
/*
This script parses the output from tshark command line
load the configuration file
*/
require('config.php');
/*
maybe we need to do some experimenting with this option…..?
*/
//stream_set_blocking(STDIN, 0);
while (false !== ($line = fgets(STDIN))) {
$fields = preg_split('/\s+/', $line);
/*
we only process the data under certain conditions
- valid mac address is submitted
- transmission method is set to &quot;rest&quot;
*/
if(filter_var($fields[1], FILTER_VALIDATE_MAC)) {
    /*
    POST a json message to a REST API
    - first assign the values collected from the probe request, to the vars
    */
    $ts     = $fields[0];
    $mac    = $fields[1];
    $rssi   = $fields[2];

    /*
    calculate the hash of the data to be submitted
    */
    $string_to_hash = $ts . &#39;|&#39; . $mac . &#39;|&#39; . $rssi . &#39;|&#39; . $drone_id . &#39;|&#39; . $public_key . &#39;|&#39; . $secret_key;
    $hash_of_string = sha1($string_to_hash);

    /*
    construct the json POST data to be submitted
    */
    $raw_data = array(&quot;ts&quot; =&gt; $ts, &quot;mac&quot; =&gt; $mac, &quot;rssi&quot; =&gt; $rssi, &quot;drone_id&quot; =&gt; $drone_id, &quot;public_key&quot; =&gt; $public_key, &quot;hash&quot; =&gt; $hash_of_string);
    $data_string = json_encode($raw_data);

    $ch = curl_init($probe_api_url);
    curl_setopt($ch, CURLOPT_CUSTOMREQUEST, &quot;POST&quot;);
    curl_setopt($ch, CURLOPT_POSTFIELDS, $data_string);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_HTTPHEADER, array(
        &#39;Content-Type: application/json&#39;,
        &#39;Content-Length: &#39; . strlen($data_string))
    );

    $result = curl_exec($ch);
}

}
?>

I have read the articles re memory consumption ( https://wiki.wireshark.org/KnownBugs/OutOfMemory )and possibly switching to “dumpcap” but i have also read several questions on here from people with similar issues saying it makes no difference. Obviously as we are using an RPI i have no way to increase the physical RAM.

If anyone can offer any suggestions on what to do… at the moment i’m not even sure how to / if i can create a log file to find out whats happening i would really appreciate it.

In order to finish my study i need the RPI to run for around 4 days.. preferably a full week. At the moment i am having to reboot it 2 -3 times a day :-(

Many Thanks and apologies for such a long post.

out-of-memory crash tshark linux

asked 16 Jun ‘16, 12:11

Pigsfoot
6●2●2●4
accept rate: 0%

edited 16 Jun ‘16, 13:01

grahamb ♦
19.8k●3●30●206

Okay, a lot of context, but what you basically say is that you loose reception on your wireless interface, which you can only recover by rebooting the Pi.

So, have a closer look at the wireless interface. Does the driver give out? Is it modular? Can you reload it?

(16 Jun ‘16, 14:59) Jaap ♦

Hi thanks for the reply. I can’t be 100% certain but I don’t think the issue is related to the adaptor or loosing reception. When we stop receiving data … I. E. The rpi stops sending data to to our Web portal I have dropped and brought up the wlan interface and it makes no difference.

I have read more about simular problems and it’s seems the issue that tshark may not be the right tool to use for long term scans when handling large amounts of data.

When we have managed to collect a good day of data, with several reboots we are seeing around 20,000 individual mac addresses… some of which multiple time as they go back and forth. As you can see from the script we run we send this data in real time to our Web server with 3 main segments. Mac address, rssi and a time stamp.

I hope this provides some more insight.

Thanks

(16 Jun ‘16, 15:16) Pigsfoot

TP-Link TL-WN722N USB WiFi Dongle

I have several of these on an embedded Linux system to sniff channels 1/6/11, with the goal of 24/7 capture on a ring buffer with dumpcap. However, they routinely hang. Only recovery for me is to power cycle the device or remove/reinsert the USB adapter. However, I reboot as I also feed the data into a network manager and that likes to have the interface index constant, and resetting the USB device will index the SNMP interface index. So reboot it is…

I found Intel-based devices to have slightly better drivers than the ARM stuff.

For me, I put the interface into monitor mode then monitor the number of packets with ifconfig. Run ifconfig a couple of times and when the Rx packets does not change, it’s hung. I wrote a shell script to automate this, but then my box can reset a lot. So I live without the data if it hangs and only use it as a best-effort back up source :(

(16 Jun ‘16, 15:27) Bob Jones

@Pigsfoot: You’re jumping to conclusions. ifdown/ifup doesn’t help if the device or driver is in trouble. These are hardware/software underneath the interface you bring down and up. It won’t necessary fix them.

Indeed dissection engines are prone to Out-Of-Memory (OOM) because they keep state. Using -1 option should alleviate that. If this is indeed a problem then should be able to pick up on that by monitoring memory usage during operations at daytime.

Setup the kernel OOM killer. That would kill tshark if it consumes ever more memory.

If it’s tshark causing problems you should simply change your script. Have it relaunch every hour, stopping it from causing trouble.

So there are may options available, if your analysis is right.

(16 Jun ‘16, 22:55) Jaap ♦

@Jaap: Thanks for the feedback, much appreciated. Thinking about what your saying does make some sense because when the web portal reports its not receiving any traffic i have logged into the RPI as per my original comment, checked the memory use and it would appear that i haven’t ran out, i also restarted my capture script which didn’t make any difference which kind of suggests it’s not a memory issue at all.

I will however add the -1 option to my command line and also as suggested set up the OOM killer kernel just to see if it helps. I still think the issue is attributed to the amount of traffic being processed because on a quiet day / night time we have no issues, i just need to work out which bit is causing the problem :-). On the pus side the issue happens all the time during the day so i will know if its fixed it or not pretty quickly. If it continues to happen i will have to look into trying to do something with the wlan…. any suggestions ?

Thanks

(17 Jun ‘16, 01:26) Pigsfoot

The RPI has actually stopped working this morning so i have just ran the following and and received the displayed responses

[email protected]:~# free -m total used free shared buffers cached Mem: 744 245 499 5 9 168 -/+ buffers/cache: 67 677 Swap: 0 0 0

ps aux | grep tshark root 362 0.0 7.1 97632 54544 ? S Jun16 0:17 /usr/bin/tshark -l -I -i wlan3 -T fields -e frame.time_epoch -e wlan.sa -e radiotap.dbm_antsignal -e wlan_mgt.ssid -E occurrence f type mgt subtype probe-req root 363 0.0 1.7 37436 13076 ? S Jun16 0:18 /usr/bin/php /opt/dronecapture/tsharkparser.php root 2823 0.0 0.0 2060 548 pts/0 S+ 08:08 0:00 grep tshark

Also

systemctl status dronecapture.service -l â dronecapture.service - Probe Request capture tool Loaded: loaded (/lib/systemd/system/dronecapture.service; enabled) Active: active (running) since Thu 2016-06-16 19:31:18 UTC; 12h ago Process: 351 ExecStart=/opt/dronecapture/php-tshark.sh (code=exited, status=0/SUCCESS) CGroup: /system.slice/dronecapture.service ââ362 /usr/bin/tshark -l -I -i wlan3 -T fields -e frame.time_epoch -e wlan.sa -e radiotap.dbm_antsignal -e wlan_mgt.ssid -E occurrence f type mgt subtype probe-req ââ363 /usr/bin/php /opt/dronecapture/tsharkparser.php ââ418 /usr/bin/dumpcap -n -i wlan3 -f type mgt subtype probe-req -I -Z none

Jun 16 19:31:18 DroneID23 systemd[1]: Started Probe Request capture tool. Jun 16 19:31:23 DroneID23 php-tshark.sh[351]: tshark: Lua: Error during loading: Jun 16 19:31:23 DroneID23 php-tshark.sh[351]: [string "/usr/share/wireshark/init.lua"]:46: dofile has been disabled due to running Wireshark as superuser. See http://wiki.wireshark.org/CaptureSetup/CapturePrivileges for help in running Wireshark as an unprivileged user. Jun 16 19:31:24 DroneID23 php-tshark.sh[351]: Running as user "root" and group "root". This could be dangerous. Jun 16 19:31:25 DroneID23 php-tshark.sh[351]: Capturing on 'wlan3'

Finally i ran - “systemctl restart dronecapture.service” which restarts the capture service and unfortunately it made no difference. The only way to bring the RPI back online is to reboot it.

(17 Jun ‘16, 01:27) Pigsfoot

OKe, more post-mortem to do. Once the system fails, don’t simply restart dronecapture, start looking at the wlan3 device. Look at system logs, see if anything pops. You could already look into the driver, see what options it provides to investigate further, and what’s published on this combination on the PI2.

PS: Did you overclock this Pi? Might want to be careful with that as well.

(17 Jun ‘16, 04:55) Jaap ♦

showing 5 of 7 show 2 more comments

2 Answers:

I think your system is failing due to hardware issues. It is likely throughput dependent: we can guess that whatever 'event' or specific scenario is causing the USB wifi adapter to fail is more likely to occur when more traffic is passed. I also recommend using tcpdump or dumpcap as the long term traffic collector, not one of the *sharks.

I have had better luck with these USB devices to do what you want:

Bus 002 Device 004: ID 0846:9012 NetGear, Inc. WNDA4100 802.11abgn 3x3:3 [Ralink RT3573]

In addition, they are more capable than the TP Link, but they cost more - I recall about $US40 on amazon.com, compared to $US13 for the TPLink. Make sure you use a recent kernel to get a good Ralink driver, but I don't know how that will work with the ARM chip. I moved away from ARM for these very reasons - to improve the quality of the Wifi captures on Linux. Note that these still crash, but I can get a couple of weeks out of them, instead of hours to days for the TP Link.

Example of system with Ralink wifi chips for packet capture:

alt text

answered 17 Jun '16, 02:57

Bob Jones
1.0k●2●5●15
accept rate: 21%

edited 17 Jun '16, 05:15

Hi Bob, That's very helpful thank you. So are you suggesting do away with the RPI and move to an Intel platform? I will get one of the WiFi adapters suggested and try that to see what difference it makes on the RPI.

I have had a look at a site i use and they offer a few Intel based alternatives to the RPI - http://uk.farnell.com/webapp/wcs/stores/servlet/Search?catalogId=15001&langId=44&storeId=10151&categoryId=700000005178&showResults=true&aa=true&sf=731&pf=110181927

Is there one there you would recommend or do you have a preferred platform you have tried?

One other question... if the issue is hardware related would having a script to drop and bring back up the wlan adapter on a regular basis make any difference? If not is there anything i can do short term while i try a different hardware platform.

Thanks

(17 Jun '16, 04:49) Pigsfoot

Is there one there you would recommend or do you have a preferred platform you have tried?

I have tried the Intel NUC and the Compulab stuff. I currently use the Compulab gear for 'spot' wifi captures with three+ USB NICs and have these distributed around the globe where I need them. I would attach a picture if I could, but I cannot to a comment.

I have had mixed success with cycling the adapter - I tried:

ifconfig wlanX down ifconfig wlanx up

and sometimes it works, but not always. There are several cases:

No effect
Device comes back in monitor mode, but not in promiscuous mode anymore (i.e. only see broadcast/multicast now, until reboot)
Comes back fully until it crashes again...

Result 2 is worst case - looks like it is up but it really isn't.

So I automated the checking of the adapters through ifconfig with cron/bash script but only reliable way to recover is a reboot. However, this was not true for the TP Link: a reboot did not always correct the issue - on some occasions I need to power cycle the USB device so I need to pull it from the socket, or kill power to the USB interface somehow. I would bet different systems handle power to usb bus differently.

I have Raspberry PIs, Udoo boards, Beagle boards. All cool, all cheap, but to get latest kernel with latest WiFi updates to improve stability, I moved to i386 embedded devices. YMMV.

(17 Jun '16, 05:08) Bob Jones

@Bob Jones, the problem with pictures in comments is not so much how to post them but that in Comments, they are not auto-sized like in Questions and Answers so if you post them in the original (high) resolution, they make the whole page look ugly.

But you can edit your original answer and add the picture there. Or (pssst!) you can post another Answer with a picture in it and then convert it into a comment to another Answer - your karma should already allow that at least for your own Answers, but please first check whether the option "more -> convert to comment" is available for your existing Answer.

(17 Jun '16, 05:44) sindy

First things first... awesome forum... getting some great help on here. Thanks guys.

To start off with.. Nope the RPI 2b is straight out of the box.. not overclocked at all.

The RPI has just stopped working again so before a reboot i ran a couple of commands

This is where i start to suffer... I'm not really sure what to look for or how to get the Wlan logs...

I have just ran the following ... not sure if it is of any help

[email protected]:~# dmesg | grep -i wlan3
[    5.539336] ath9k_htc 1-1.3:1.0 wlan3: renamed from wlan0
[    5.608374] systemd-udevd[160]: renamed network interface wlan0 to wlan3
[   21.144337] device wlan3 entered promiscuous mode
[    2.273783] usb 1-1.3: Product: USB2.0 WLAN
[    2.275528] usb 1-1.3: Manufacturer: ATHEROS
[    2.277204] usb 1-1.3: SerialNumber: 12345
[    4.513412] usb 1-1.3: ath9k_htc: Firmware htc_9271.fw requested
[    4.515626] usbcore: registered new interface driver ath9k_htc
[    4.818512] usb 1-1.3: ath9k_htc: Transferred FW: htc_9271.fw, size: 50980
Also, :- ifconfig shows this
wlan3     Link encap:UNSPEC  HWaddr F4-F2-6D-13-EC-41-30-30-00-00-00-00-00-00-00-00
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:4945759 errors:0 dropped:8 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1181209635 (1.1 GiB)  TX bytes:0 (0.0 B)

(17 Jun '16, 06:33) Pigsfoot

I also ran this

[email protected]:~# dmesg | grep -i usb
[    0.285161] usbcore: registered new interface driver usbfs
[    0.285293] usbcore: registered new interface driver hub
[    0.285461] usbcore: registered new device driver usb
[    0.414090] usbcore: registered new interface driver smsc95xx
[    0.843470] dwc_otg bcm2708_usb: DWC OTG Controller
[    0.845027] dwc_otg bcm2708_usb: new USB bus registered, assigned bus number 1
[    0.846612] dwc_otg bcm2708_usb: irq 32, io mem 0x00000000
[    0.851416] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    0.852954] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    0.854479] usb usb1: Product: DWC OTG Controller
[    0.855967] usb usb1: Manufacturer: Linux 3.18.16-v7 dwc_otg_hcd
[    0.857493] usb usb1: SerialNumber: bcm2708_usb
[    0.859878] hub 1-0:1.0: USB hub found
[    0.863722] usbcore: registered new interface driver usb-storage
[    0.917692] usbcore: registered new interface driver usbhid
[    0.919294] usbhid: USB HID core driver
[    1.257587] usb 1-1: new high-speed USB device number 2 using dwc_otg
[    1.457889] usb 1-1: New USB device found, idVendor=0424, idProduct=9514
[    1.459777] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    1.462411] hub 1-1:1.0: USB hub found
[    1.737681] usb 1-1.1: new high-speed USB device number 3 using dwc_otg
[    1.840097] usb 1-1.1: New USB device found, idVendor=0424, idProduct=ec00
[    1.840110] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    1.911682] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1, smsc95xx USB 2.0 Ethernet, b8:27:eb:6d:19:c4
[    2.147844] usb 1-1.3: new high-speed USB device number 4 using dwc_otg
[    2.268382] usb 1-1.3: New USB device found, idVendor=0cf3, idProduct=9271
[    2.270269] usb 1-1.3: New USB device strings: Mfr=16, Product=32, SerialNumber=48
[    2.273783] usb 1-1.3: Product: USB2.0 WLAN
[    2.275528] usb 1-1.3: Manufacturer: ATHEROS
[    2.277204] usb 1-1.3: SerialNumber: 12345
[    4.513412] usb 1-1.3: ath9k_htc: Firmware htc_9271.fw requested
[    4.515626] usbcore: registered new interface driver ath9k_htc
[    4.818512] usb 1-1.3: ath9k_htc: Transferred FW: htc_9271.fw, size: 50980
[email protected]:~#

(17 Jun '16, 06:35) Pigsfoot

Have you polled StackExchange?, because this is getting a bit out of scope.

(17 Jun '16, 07:18) Jaap ♦

Try:

iw dev

iwconfig

ifconfig (run multiple times, see if Rx packets is changing)

With WiFi, usually 2.4GHz is very busy so there are always networks out there and they beacon (usually) at ~100ms. So I typically expect to always see packets; even if the channel happens to be perfectly clear of networks, you should still catch probes now and again.

(17 Jun '16, 07:56) Bob Jones

Hi, Based on suggestions of where the issue could be I have been looking into different Wireless adapters to try, I have an Alfa awus036nh, would that be better than the current TP-Link ? The Alfa range seems to be very popular in network sniffing and well supported for monitor mode.

Can anyone point me in the right direction on where to look within Linux to see if it is the wireless adapter that has stopped working.

Thanks

(18 Jun '16, 08:04) Pigsfoot

Hi @bob-jones

Thanks for the feedback, thats just what i needed to know. I have done as suggested and when the drone goes "off line" as in the web portal reports its not receiving any traffic from the RPI i checked it and ran the ifconfig command several times and the RX packets are not changing. However if i do the same when it is working the number of packets change every time i run the command, literally one after the other.

I'm going to the Alfa AWUS036NH tomorrow and swap it out from the TP Link. From what i have read thats the better adapter and it uses a different chipset, on the plus side it shouldn't take long to see if it makes a difference. I have tried it on a spare RPI i have at home and it is recognized straight away when i plug it in so fingers crossed.

Thanks

(19 Jun '16, 07:33) Pigsfoot

This is probably best continued on the RPi stack exchange site as @Jaap says. It's really a hardware/OS issue, not really a packet capture issue. I have better results with the RT chipsets so maybe the performance will better when you switch. Note they have issues too, so no guarantees.

Some points to move forward:

https://lists.ath9k.org/mailman/listinfo/ath9k-devel https://wireless.wiki.kernel.org/en/users/drivers/ath9k https://wireless.wiki.kernel.org/en/users/drivers/ath9k/debug

(19 Jun '16, 10:46) Bob Jones

showing 5 of 10 show 5 more comments

0	Thanks for all your help guys... at least i know where to concentrate my efforts now. answered 19 Jun '16, 14:07 Pigsfoot 6●2●2●4 accept rate: 0%