how do I configure dumpcap from tshark while pipeing the tshark output to my program?

Question

I'm trying to configure tshark to instruct dumpcap to create a raw network data capture ring buffer in ram disk while pipeing the filtered packets output by tshark to my program. It worked fine UNTIL I tried to set up a ring buffer in ram. I now get this error message:

tshark: Read filters aren't supported when capturing and saving the captured packets.

Here's my tshark command line:

/usr/bin/tshark -nn -t e -i wlan0 -R "wlan.fc.type==0 and wlan.fc.subtype==4" -e frame.time -e wlan.sa -e radiotap.dbm_antsignal -Tfields -E separator=, -l -b files:10 -b filesize:10000 -w /var/tmp/wlan0_capture | <my_program>.sh &

I've set wlan0 into monitormode. I'm running all the latest code on Raspian.

How do I fix this ? Thank you in advance.

Answer 1

0

Read filters are only supported when reading a capture from file. For capture filtering, you need to use a capture filter which uses the limited pcap filter format rather than the more expressive wireshark display filter format.

Have a look at the pcap-filter manual page. You probably need to replace -R "wlan.fc.type == 0 and wlan.fc.subtype == 4" by:

-f 'type 0 subtype 4`

answered 15 Dec '14, 06:17

Lekensteyn
2.2k●3●7●24
accept rate: 30%

Thank you, but the -R format works so long as I don't use the -b and -w options. Is there a performance difference between -R and -f ?

How do I tell dumpcap, through the tshark command line, to create a circular buffer at a particular location ? Maybe tshark is getting confused by -w, which I'm trying to use to create the capture buffer, rather than the output from tshark. The dumpcap spec says to use -w, and I'm assuming tshark passes commands through to dumpcap.

(15 Dec '14, 07:08) NewtownGuy

@NewtownGuy

A display filter with the -R option is incompatible with -w, that's why you have to use the capture filter option -f and adapt the syntax accordingly.

For a circular buffer use the -b option, and as it is only usable with the -w option, again this is not compatible with a -R display filter, you have to use -f capture filter.

(15 Dec '14, 07:18) grahamb ♦

Thank you. How does tshark know that -w /path/filename, when used with -b filesize:10000 -b file:10, applies to dumpcap and not its outfile ?

(15 Dec '14, 08:15) NewtownGuy

Here's my revised command line using -f, -b and -w, but it does not work. It creates a 160-byte binary file at /var/tmp/capture_wlan0_xxx, but it's static. The manual for tshark says the -b and -w options control the (output?) file that tshark creates, but I want to control the file that dumpcap creates and feeds to tshark. I want to pipe the filtered output from tshark to myscript.sh, so if dumpcap -f does the filtering, why do I need tshark ?

/usr/bin/tshark -n -t e -i wlan0 -f "type 0 subtype 4" -e frame.time -e wlan.sa -e radiotap.dbm_antsignal -T fields -E separator=, -l -b files:10 -b filesize:10000 -w /var/tmp/capture_wlan0 | /path/myscript.sh &

How should I fix this ?

Thank you.

(15 Dec '14, 15:06) NewtownGuy

What version of tshark are you using? Using multiple files and -T fields output works for me with 1.99-2.

You need tshark to dissect the data and produce the fields you are piping to myscript.sh. Dumpcap handles the capturing and file writing, and it pipes the data to tshark which then dissects the data and outputs the text.

You seem to be confused by the filtering. You are using capture filtering (performed by dumpcap) using the -f option that filters what is fed to the dissection engine (libwireshark, which is part of tshark and Wireshark) and what is saved into the capture files, and then output field selection by tshark using the -T fields -e xxx options.

There are also display filters (tshark -R and -Y options) that restrict output, but do not affect what is captured.

See this question on the differences between capture and display filters for more info.

(16 Dec '14, 05:28) grahamb ♦

I'm running tshark 1.8.2-5wheez. I looked for the dumpcap version, but it's not listed in dpkg --list, so it must be embedded in tshark.

I'm confused by the -w option because tshark uses it to write its output file, but dumpcap uses it to write its capture file. While I don't need tshark to write its output to a file (only a pipe), it's not clear to me if there's a way to write a tshark command line that pipes its output to a script, while using -w to control the capture file created by dumpcap. It seems to me that tshark needs one parameter to write its output file, and a different parameter to tell dumpcap where to write its capture file that is fed into tshark.

I'm also confused by dumpcap filtering. Unless the cpu is awfully fast, I don't see how it can process a high data rate without buffering to a file first, so it doesn't drop packets. I'm seeing occasional dropped packets now on my wireless interfaces, without any filtering.

If I were able to get dumpcap to write a filtered capture file to ram disk, and thus greatly reduce how much is written to disk, can tshark format it and pipe it to my script in real time ?

(17 Dec '14, 09:14) NewtownGuy

Your answer has been converted to a comment as that's how this site works. Please read the FAQ for more information.

You're running a somewhat old version, current stable is 1.12.2, there may be some issues there. Certainly what you are attempting to do works for me using 1.12.2.

When capturing, tshark (and Wireshark) start dumpcap to make the capture and write out the capture files. dumpcap also feeds the packets to tshark (and Wireshark) over a pipe. tshark (and Wireshark) then process the packets for output at their own rate, possibly leading to lag behind the actual capture.

A capture filter (or dumpcap filtering as you call it) is a high performance filter using bpf. The high performance requirements dictate the less extensive filtering capabilities. The packets are buffered by the kernel in the capturing mechanism and packets can be dropped, usually when using very high line rate, e.g. 10GBe. The fact that line rates exceed disk I/O rates is one reason for capture filtering so that the torrent of packets from a high line rate can be reduced to something that the disk I/O can handle.

Dumpcap does write the capture filtered file to disk, but there is no mechanism for tshark (or Wireshark) to read that capture file, instead, as I've described, they consume packets from dumpcap via a pipe.

Hopefully you are aware that running tshark continuously in this manner will cause tshark to eventually run out of memory due to maintain state about conversations?

(17 Dec '14, 10:34) grahamb ♦

showing 5 of 7 show 2 more comments

Answer 2

If you don't mind a lag between when dumpcap finishes writing to a particular file of the ringbuffer and when tshark processes that file, then the following script might be of use to you. I don't claim to be a script expert, so improvements are certainly possible.

This script was tested using Wireshark 1.12.1 command-line tools, so it may not run with your version of Wiresshark. For example, you may need to replace the "-Y" option with "-R".

#!/bin/sh
# This script starts dumpcap with ring buffer settings and then feeds a
# completed capture file to tshark for further processing.
#
# The tshark output will lag because it must wait for a capture file to be
# fully written first.
#
# When dumpcap terminates, the last capture file in progress will be processed
# by tshark.
Check usage
if (( $# < 1 ))
then
<tab>echo "Usage: $0 < prefix >"
<tab>exit 0
fi
Cleanup any old lingering files or processes …
killall dumpcap tshark &> /dev/null
###################################
Initialize stuff, edit as needed.
prefix=$1_date +%s
dumpcap-specific
interface="eth0"
capfilter="icmp"
ringbuffer="-b files:10 -b filesize:10"
tshark-specific
filter="icmp"
fields="-T fields -E separator=, -e frame.number -e frame.time_epoch"
########################################
Start dumpcap and save its process ID
dumpcap -q -i $interface -f "$capfilter" $ringbuffer -w $prefix &
sleep 1
dcpid=pidof dumpcap
pid=$dcpid
let count=1
let next=$(( (count + 1) % 100000))
curfile=$( ls -1 printf ${prefix}_%05d_* $count )
#################################################
Process one complete ring buffer file at a time
while [[ $dcpid = $pid ]]
do
sleep 1
# See if dumpcap is done with the current file
nextfile=$( ls -1 printf ${prefix}_%05d_* $next 2> /dev/null )
if [[ -f $nextfile ]]
then
tshark -r $curfile -Y "$filter" $fields | ./myscript.sh
curfile=$nextfile
let count=$next
let next=$(( (count + 1) % 100000))
fi
pid=pidof dumpcap
done
tshark -r $curfile -Y "$filter" $fields | ./myscript.sh
Final cleanup (useful during testing)
killall dumpcap tshark &> /dev/null