This is a static archive of our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

Tshark Uses All Memory on Mavericks Triggering Out Of Application Memory Errors

0

I use tshark extensively to process pcap/pcapng files captures with wireshark.

While tshark has always been a nasty memory hog, prior to Mavericks, this has not been an issue.

Running tshark against even moderate captures files (<1GB) will trigger all kinds of nasty Mavericks specific memory issues.

For example, on Mavericks Quad Core MacPro with 22GB physical memory, processing a single 800MB for a single ip address filer will:

  • drive tshark memory usage to 18GB
  • allocate a total of 80GB of virtual memory
  • trigger memory compression of other apps
  • lock tshark on an IO call, so it never actually finished
  • trigger a "Out of Application memory error" where Mavericks wants you to kill apps
  • killing tshark will return the memory stage of the system back to normal

Is there an upcoming Mavericks specific build that may address this. At this stage, tshark is no longer usable on Mavericks.

asked 08 Apr '14, 18:24

mhpcto's gravatar image

mhpcto
1●1●1●2
accept rate: 0%

edited 10 Apr '14, 11:50

Hadriel's gravatar image

Hadriel
2.7k●2●9●39

What exact release of wireshark? Can you paste the output of "tshark -v" here?

(08 Apr '14, 19:32) Hadriel

Oh, and what command-line options are you giving tshark?

(08 Apr '14, 19:32) Hadriel

kumarogg:WiFi-Tests roland$ tshark -v TShark 1.10.3 (SVN Rev 53022 from /trunk-1.10)

Copyright 1998-2013 Gerald Combs [email protected] and contributors. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Compiled (64-bit) with GLib 2.36.0, with libpcap, with libz 1.2.3, without POSIX capabilities, without libnl, with SMI 0.4.8, without c-ares, without ADNS, with Lua 5.1, without Python, with GnuTLS 2.12.19, with Gcrypt 1.5.0, with MIT Kerberos, with GeoIP.

Running on Mac OS X 10.9.2, build 13C64 (Darwin 13.1.0), with locale en_US.UTF-8, with libpcap version 1.3.0 - Apple version 41, with libz 1.2.5. Intel(R) Xeon(R) CPU E5620 @ 2.40GHz

Built using llvm-gcc 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.9.00).

Typically, I am extracting a range of stats, but the issue shows up even in the most simple case of a single IP address and transfer stats:

tshark -q -z 'io,stat,0.1,ip.addr==172.24.110.175' -r UDP_Sequence_10.pcapng > UDP_Sequence_10.txt

(10 Apr '14, 10:48) mhpcto

Can you try it with 1.10.6? I have no idea if it would help, but there were bug fixes between 1.10.3 and 1.10.6.

Also, you said they were "Mavericks specific" - do you mean running the exact same Wireshark version on a non-Mavericks OSX doesn't have this problem?

(10 Apr '14, 11:50) Hadriel

Yes, this is specific to Mavericks. I have been using the same version of tshark on both OSX 10.6 and 10.8 without the runaway memory consumption.

Using 1.10.6 only slightly changes behavior:

  1. tshark peak memory consumption is around 6GB (18GB with 1.10.3)
  2. peak memory is 68GB (80GB with 1.10.3)
  3. memory use increments slower (by about 50%)
  4. kernel_task still peaks at 10GB
  5. memory compression still kicks in
  6. memory pressure still eventually reaches the red-zone and the system grinds to a halt
  7. tshark itself still eventually goes down to 0% CPU and never completes and must be killed to restore

What exacerbates the problem under Mavericks is the new "App Nap" feature. Due to tshark memory usage ballooning to 18GB, Mavericks will start to suspend a lot of apps and many of them (firefox, for example) will not gracefully wake.

As a workaround I can run tshark in a Parallels VM (allocated 4 cores and 4GB) running Debian, Fedora or CentOS and tshark will cleanly and quickly process the input file.

(17 Apr '14, 14:14) mhpcto

Sounds like a bug. I suggest you submit a bug at bugs.wireshark.org, with all of the above information. And also attach an example pcap file if you're able to.

(19 Apr '14, 10:40) Hadriel

To further clarify that this is tshark only.

I can read the same pcapng file into wireshark 1.10.6 and that has no measurable impact on the memory footprint. wireshark-bin will allocate a bit under 1GB and fluctuate around that number regardless of how many filters I use, stats and IO graphs I generate, etc.

Does not increase what Apple calls memory pressure not does it affect app memory.

(21 Apr '14, 10:47) mhpcto
showing 5 of 7 show 2 more comments

One Answer:

0

While this behavior could be a bug, I'm not sure it is one. Wireshark and tshark are known to consume large amount of memory with large capture files.

http://wiki.wireshark.org/KnownBugs/OutOfMemory

However, a ratio 1:80 (1 Gig Capture file -> 80 Gig RAM) look more like a big than anything else.

Solution: Here is what you can do while the (possible) bug is under investigation. Besides your workaround with the virtual machine, you can run tcpdump to pre-filter the capture file.

tcpdump -nr input.pcap -w output.pcap "host x.x.x.x or port y.y.y.y"

Use whatever capture filter you may need to pre-filter the capture file and only then run tshark for statistics and/or other analysis tasks. That should at least make tshark again usable for you.

Regards
Kurt

answered 19 Apr '14, 15:08

Kurt%20Knochner's gravatar image

Kurt Knochner ♦
24.8k●10●39●237
accept rate: 15%

Unfortunately, I already tried this by:

Using a minimal complexity capture (single IP source address filter)
Using a shortened capture file (10MB instead of the 800MB original)
Simplifying the io stat expression to once per second on the single IP address

The results are consistent in that:

Size of the file doesn't matter
Samples extracted (1s vs 0.1s sampling) doesn't matter.

The following example is for sampling a 10MB PCAPNG file at 1s. I am showing the Memory allocation for tshark-bin and kernal_task at an interval of 5 seconds in GB:

Tshark - kernel_task

0 - 1.7
0.4 - 1.7
4.0 - 1.7
6.6 - 1.7
10.3 - 1.7
10.7 - 1.7
10.1 - 1.7
9.5 - 1.7
8.9 - 1.7
7.5 - 5.7
6.9 - 5.7
5.8 - 5.7
5.2 - 5.7
4.6 - 5.7
4.1 - 9.7
3.1 - 9.7
2.8 - 9.7
2.9 - 9.7
3.1 - 9.8
3.1 - 9.8
....
2.0 - 9.8

Repeated runs show that tshark, on this file, can peak at up to 14GB. This does not caus an increase in memory pressure. Memory pressure starts increasing as soon as memory growth of tshark and kernel_task invert. In other words, as soon as tshark memory allocation decreases and kernel_task memory increases, memory pressure shoots up, virtuall memory allocation and compressed memory start increasing until memory pressure reaches the red zoneand starts ti impact the whoile system.

(21 Apr '14, 17:00) mhpcto

O.K. now it sounds like a problem with the tshark io stats function.

Can you please run the following command and check the RAM usage?

tshark -nr input.pcapng -q

while input.pcap is the already 'pre-filtered' capture file.

I just want to check if tshark (actually the dissector for the protocol) is exhausting the RAM or if it's the io stats function. If you don't see the same RAM usage with the above command, it's a problem with the io stats function. If you see the same problem, it's probably related to the dissector of the protocol.

(22 Apr '14, 01:48) Kurt Knochner ♦

Running tshark with the -qnr options, on any size capture file, does not lead to the described symptoms. Memory allocation is constant at ~50MB.

Using tshark to capture live traffic, with or w/o capture filters, works fine as well, with a stable memory footprint. Simple, non io stat operations, like "-n -r /tmp/xxxf.pcap -T fields -e ip.src" are also stable.

I want to throw in a piece of tangential information:

I downloaded and build tshark 1.11.4. to build it on 10.8 and 10.9 and profile it. Didn't get around to that yet, but what I discovered is that 1.11.4 does not handle the simple test I used above for 1.10.6 - "-n -r /tmp/xxxf.pcap -T fields -e ip.src" will not terminate and slowly increase memory by approx 1MB/sec. 1.10.6 parses the same 3MB capture in about 5sec. Guess I'll grab 1.10.6 instead.

(22 Apr '14, 11:52) mhpcto

O.K. sounds like a bug in the io stats function of tshark. As @Hadriel already mentioned: Please file a bug report at https://bugs.wireshark.org

(23 Apr '14, 12:00) Kurt Knochner ♦

Bug #10068 filed.

(05 May '14, 11:47) mhpcto