Tshark stops output redirection on Linux

Question

Hello!

I'm trying to capture traffic with applying tshark filters in realtime.

#Capturing command is the following:
tshark -i eth6 -i eth7 -R '(tcp.analysis.retransmission or tcp.analysis.fast_retransmission or tcp.analysis.duplicate_ack_frame)' -Tfields -Eseparator="|" -Eoccurrence=l -e frame.time -e ip.src -e tcp.srcport -e ip.dst -e tcp.dstport -e expert | sed "..." > /tmp/retransmissions.txt

After the some time odd situation occurs: file /tmp/retransmissions.txt stop grows, but wireshark temporary file still grows (looks like everything is fine and that traffic still captures). If I will start additional tshark process with the same filters (without redirection to file), I will see that retransmissions are present, but 1st process of the tshark will not redirect to output these packets.

So the situation looks like redirection unexpectedly stops its work. Every necessary process runs in this time:

# ps aux |grep tshark root 21992 4.2 1.0 493688 304092 pts/1 S 09:56 1:39 /usr/bin/tshark -i eth6 -i eth7 -R (tcp.analysis.retransmission or tcp.analysis.duplicate_ack_frame) -Tfields -Eseparator | -Eoccurrence l -e frame.time -e ip.src -e tcp.srcport -e ip.dst -e tcp.dstport -e expert ps aux |grep dumpcap root 22008 3.0 0.0 202288 26428 pts/1 Sl 09:56 1:48 /usr/bin/dumpcap -t -n -i eth6 -i eth7 -Z none free total used free shared buffers cached Mem: 28729376 3672100 25057276 0 192384 1397272 -/+ buffers/cache: 2082444 26646932 Swap: 6143992 0 6143992 tshark -v TShark 1.10.3 (SVN Rev Unknown from unknown) cat /etc/redhat-release

CentOS release 6.5 (Final)

Do you have any ideas why this situation may occurs and how to resolve it?

Thanks in advance!

Answer 1

2

tshark output is buffered. Please try tshark option -l and check if that helps. Same for sed, option -u.

Regards
Kurt

answered 03 Dec '13, 01:44

Kurt Knochner ♦
24.8k●10●39●237
accept rate: 15%

edited 03 Dec '13, 01:49

Kurt,

I have tried this solution, unfortunately problem has occurred again.

(03 Dec '13, 02:59) mrav

O.K. then please do the following:

As soon as the output stalls, run the following commands

strace -r -tt -T -s 1024 -p <pid of tshark> -f -o /tmp/tshark.strace    
strace -r -tt -T -s 1024 -p  <pid of sed> -f -o /tmp/sed.strace

Then post the output files

(03 Dec '13, 03:13) Kurt Knochner ♦

cat tshark.strace

27203      0.000000 write(2, "\r128413 ", 8 <unfinished ...>
27203    835.667210 +++ killed by SIGKILL +++

cat sed.starce

27204      0.000000 read(0, "", 4096)   = 0 <799.749215>
27204    799.749351 close(0)            = 0 <0.000040>
27204      0.000118 munmap(0x7f4405f52000, 4096) = 0 <0.000024>
27204      0.000081 close(1)            = 0 <0.000048>
27204      0.000113 munmap(0x7f4405f4e000, 4096) = 0 <0.000065>
27204      0.000122 close(2)            = 0 <0.000013>
27204      0.000087 exit_group(0)       = ?

SIGKILL and close (0) has appeared in strace files after I have stopped the capturing script.

(03 Dec '13, 05:11) mrav

27203 0.000000 write(2, "\r128413 ", 8 <unfinished ...=""> 27203 835.667210 +++ killed by SIGKILL +++

is that the only line in tshark.trace?

BTW: What is the output of the following commands, while the output stalls

df -h
lsof -n | egrep '(tshark|dumpcap)'

(03 Dec '13, 05:51) Kurt Knochner ♦

Correct, only this line, which is not gave us any useful information..unfortunately.

Will try additionally to run strace from the beginning, when tshark works correctly .Maybe this will give additional information.

Disk space is present, while the output stalls. Anyway will try again to reproduce an issue and check the command output.

Many thanks for your help!

(03 Dec '13, 05:57) mrav

Will try additionally to run strace from the beginning, when tshark works correctly .Maybe this will give additional information.

that will generate way too much data....

Please run strace on dumpcap while the output stalls.

(03 Dec '13, 06:59) Kurt Knochner ♦

Output of commands is the following:

lsof -n |egrep '(tshark|dumpcap)'

tail       2984      root    3r      REG                8,5 1061880053         18 /tmp/tshark.strace
tshark    28443      root  cwd       DIR                8,2       4096     524289 /root
tshark    28443      root  rtd       DIR                8,2       4096          2 /
tshark    28443      root  txt       REG                8,2    1227674    1709133 /usr/bin/tshark
tshark    28443      root  mem       REG                8,2      12504    1704720     /usr/lib64/gconv/IBM437.so
tshark    28443      root  mem       REG                8,2      26060    1704864 /usr/lib64/gconv/gconv-modules.cache
tshark    28443      root  mem       REG                8,2   99158576    1716876 /usr/lib/locale/locale-archive
tshark    28443      root  mem       REG                8,2      65928     655653 /lib64/libnss_files-2.12.so
tshark    28443      root  mem       REG                8,2     178014    1709122 /usr/lib/wireshark/plugins/1.10.3/gryphon.so
tshark    28443      root  mem       REG                8,2    1035851    1709127 /usr/lib/wireshark/plugins/1.10.3/profinet.so
tshark    28443      root  mem       REG                8,2     186948    1709119 /usr/lib/wireshark/plugins/1.10.3/asn1.so
tshark    28443      root  mem       REG                8,2     260738    1709125 /usr/lib/wireshark/plugins/1.10.3/mate.so
tshark    28443      root  mem       REG                8,2      34247    1709128 /usr/lib/wireshark/plugins/1.10.3/stats_tree.so
tshark    28443      root  mem       REG                8,2     167035    1709131 /usr/lib/wireshark/plugins/1.10.3/wimaxasncp.so
tshark    28443      root  mem       REG                8,2      57183    1709124 /usr/lib/wireshark/plugins/1.10.3/m2m.so
tshark    28443      root  mem       REG                8,2     141459    1709123 /usr/lib/wireshark/plugins/1.10.3/irda.so
tshark    28443      root  mem       REG                8,2    1001620    1709120 /usr/lib/wireshark/plugins/1.10.3/docsis.so
tshark    28443      root  mem       REG                8,2     178299    1709132 /usr/lib/wireshark/plugins/1.10.3/wimaxmacphy.so
tshark    28443      root  mem       REG                8,2    1709096    1709130 /usr/lib/wireshark/plugins/1.10.3/wimax.so
tshark    28443      root  mem       REG                8,2     635758    1709126 /usr/lib/wireshark/plugins/1.10.3/opcua.so
tshark    28443      root  mem       REG                8,2     252218    1709129 /usr/lib/wireshark/plugins/1.10.3/unistim.so
tshark    28443      root  mem       REG                8,2     335925    1709121 /usr/lib/wireshark/plugins/1.10.3/ethercat.so
tshark    28443      root  mem       REG                8,2      19536     655643 /lib64/libdl-2.12.so
tshark    28443      root  mem       REG                8,2      11816     655711 /lib64/libgmodule-2.0.so.0.2600.1
tshark    28443      root  mem       REG                8,2      43832     655665 /lib64/librt-2.12.so
tshark    28443      root  mem       REG                8,2      17536     655715 /lib64/libgthread-2.0.so.0.2600.1
tshark    28443      root  mem       REG                8,2    1921216     655637 /lib64/libc-2.12.so
tshark    28443      root  mem       REG                8,2     142640     655661 /lib64/libpthread-2.12.so
tshark    28443      root  mem       REG                8,2      88600     655688 /lib64/libz.so.1.2.3
tshark    28443      root  mem       REG                8,2     655691    1716807 /usr/lib/libpcap.so.1.4.0
tshark    28443      root  mem       REG                8,2     596264     655645 /lib64/libm-2.12.so
tshark    28443      root  mem       REG                8,2    1066448     655709 /lib64/libglib-2.0.so.0.2600.1
tshark    28443      root  mem       REG                8,2      58574    1708933 /usr/lib/libwsutil.so.3.0.0
tshark    28443      root  mem       REG                8,2  133618918    1709118 /usr/lib/libwireshark.so.3.1.3
tshark    28443      root  mem       REG                8,2    1291175    1708940 /usr/lib/libwiretap.so.3.0.3
tshark    28443      root  mem       REG                8,2     154520     655363 /lib64/ld-2.12.so
tshark    28443      root    0u      CHR             136,11        0t0         14 /dev/pts/11
tshark    28443      root    1w     FIFO                0,8        0t0    1727249 pipe
tshark    28443      root    2u      CHR             136,11        0t0         14 /dev/pts/11
tshark    28443      root    3r     FIFO                0,8        0t0    1727311 pipe
tshark    28443      root    4r      REG                8,5 1288595548         12 /tmp/wireshark_2_interfaces_20131203175838_Fs5DKR
dumpcap   28451      root  cwd       DIR                8,2       4096     524289 /root
dumpcap   28451      root  rtd       DIR                8,2       4096          2 /
dumpcap   28451      root  txt       REG                8,2     233105    1709134 /usr/bin/dumpcap
dumpcap   28451      root  mem       REG                8,2      65928     655653 /lib64/libnss_files-2.12.so
dumpcap   28451      root  mem       REG                0,6               1727315 socket:[1727315] (stat: No such file or directory)
dumpcap   28451      root  mem       REG                0,6               1727314 socket:[1727314] (stat: No such file or directory)
dumpcap   28451      root  mem       REG                8,2      19536     655643 /lib64/libdl-2.12.so
dumpcap   28451      root  mem       REG                8,2      11816     655711 /lib64/libgmodule-2.0.so.0.2600.1
dumpcap   28451      root  mem       REG                8,2      43832     655665 /lib64/librt-2.12.so
dumpcap   28451      root  mem       REG                8,2    1921216     655637 /lib64/libc-2.12.so
dumpcap   28451      root  mem       REG                8,2     142640     655661 /lib64/libpthread-2.12.so
dumpcap   28451      root  mem       REG                8,2      88600     655688 /lib64/libz.so.1.2.3
dumpcap   28451      root  mem       REG                8,2     655691    1716807 /usr/lib/libpcap.so.1.4.0
dumpcap   28451      root  mem       REG                8,2    1066448     655709 /lib64/libglib-2.0.so.0.2600.1
dumpcap   28451      root  mem       REG                8,2      17536     655715 /lib64/libgthread-2.0.so.0.2600.1
dumpcap   28451      root  mem       REG                8,2      58574    1708933 /usr/lib/libwsutil.so.3.0.0
dumpcap   28451      root  mem       REG                8,2     154520     655363 /lib64/ld-2.12.so
dumpcap   28451      root    0u      CHR             136,11        0t0         14 /dev/pts/11
dumpcap   28451      root    1w     FIFO                0,8        0t0    1727249 pipe
dumpcap   28451      root    2w     FIFO                0,8        0t0    1727311 pipe
dumpcap   28451      root    3u     pack            1727314        0t0        ALL type=SOCK_RAW
dumpcap   28451      root    4w     FIFO                0,8        0t0    1727311 pipe
dumpcap   28451      root    5u     pack            1727315        0t0        ALL type=SOCK_RAW
dumpcap   28451      root    6u      REG                8,5 1288595548         12 /tmp/wireshark_2_interfaces_20131203175838_Fs5DKR
strace    28847      root    3w      REG                8,5 1061880053         18 /tmp/tshark.strace

df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        32G   21G  9.8G  68% /
tmpfs            14G     0   14G   0% /dev/shm
/dev/sda1       194M   76M  108M  42% /boot
/dev/sda5        97G  2.5G   90G   3% /tmp

strace

28443      0.004057 read(4, "D\36\241ZA4\0\17S\fI|\10\0E\0\0t\372\[email protected]\[email protected]\6N\226\n\1n\204\n\1n\202N  \[email protected]\237,\304\241\5\366\256m\200\30\0\201\304q\0\0\1\1\10\n\240\265ax9\231l[\0\30\34L\0\0\0f\0\0\0,\0\0\0,\0\0\0\0LISTENER[2]APP[10100]#[email protected]\0\0\0\244\0\0\0\6\0\0\0d\0\0\0\0\0\0\0\242\354\4\0\7\211\311aB\0\0\0B\0\0\0\0\17S\fI|D\36\241ZA4\10\0E\0\[email protected]\[email protected]\6%Q\n\1n\202\n\1n\204\[email protected] \5\366\256m\237,\304\341\200\20\1>\247~\0\0\1\1\10\n9\231l^\240\265ax\0\0d\0\0\0\6\0\0\0d\0\0\0\1\0\0\0\242\354\4\0\355\225\311aB\0\0\0B\0\0\0D\36\241ZA4\0\17S\fI|\10\0E\0\0004\233\[email protected]\[email protected]\6\256\20\n\1n\204\n\1n\202N\"\271+u\223\347\365\333E\330\2\200\20\1d\304\4\0\0\1\1\10\n\240\265a|9\231la\0\0d\0\0\0\6\0\0\0x\0\0\0\0\0\0\0\242\354\4\0\365\225\311aV\0\0\0V\0\0\0\0\17S\fI|D\36\241ZA4\10\0E\0\0H\[email protected]\[email protected]\6\265\203\n\1n\202\n\1n\204\271+N\"\333E\327\356u\223\347\365\200\30\1\365Q,\0\0\1\1\10\n9\231la\240\265a\v\0\0\0\0\0\0\0e\0\0\0\0\0\0\0\0\0Ir\2\0\0x\0\0\0\6\0\0\0\244\0\0\0\1\0\0\0\242\354\4\0\322\325\311a\202\0\0\0\202\0\0\0D\36\241Z1\350\0\17S\fI|\10\0E\0\0t\33\[email protected]\[email protected]\6-\260\n\1n\204\n\1n\203N \262\232\237}\352:T\36\370x\200\30\0.\314\231\0\0\1\1\10\n\240\265a\214\32\366\340\234\0\30\34L\0\0\0f\0\0\0,\0\0\0,\0\0\0\0LISTENER[2]APP[10100]#[email protected]\0\0\0\244\0\0\0\6\0\0\0d\0\0\0\0\0\0\0\242\354\4\0\27\326\311aB\0\0\0B\0\0\0\0\17S\fI|D\36\241Z1\350\10\0E\0\[email protected]\[email protected]\6\364\212\n\1n\203\n\1n\204\262\232N T\36\370x\237}\352z\200\20\0.\257\365\0\0\1\1\10\n\32\366\341\r\240\265a\214\0\0d\0\0\0\6\0\0\0\244\0\0\0\1\0\0\0\242\354\4\0\304\335\311a\202\0\0\0\202\0\0\0D\36\241Z1\350\0\17S\fI|\10\0E\0\0t\210\[email protected]\[email protected]\6\300\322\n\1n\204\n\1n\203N\"\332\212u\213\271D*\r\245\371\200\30\1t\277\2\0\0\1\1\10\n\240\265a\216\32\366\340V\0\n\326t\0\0\0f\0\0\0,\0\0\0,\0\0\0\0LISTENER[1]APP[10102]#[email protected]\0\0\0\244\0\0\0\6\0\0\0|\0\0\0\0\0\0\0\242\354\4\0\366\335\311a\\\0\0\0\\\0\0\0\377\377\377\377\377\377\234\216\231\36\371|\10\0E\0\0N\21\355\0\0\200\02178\n\1ny\n\1n\377\0\211\0\211\0:\320\262\236\276\1\20\0\1\0\0\0\0\0\0 FHEJEOCNEIDCDBEMEMDIEDDDDFEFECAA\0\0 \0\1|\0\0\0\6\0\0\0d\0\0\0\0\0\0\0\242\354\4\0"..., 4096) = 3808 <0.000121>
28443      0.000568 read(4, "", 288)    = 0 <0.000111>
28443      0.003445 write(2, "\r257636 ", 8 <unfinished ...>

(03 Dec '13, 11:08) mrav

showing 5 of 7 show 2 more comments