tshark crashed without any reason in output log

Question

Hi all, I'm using tshark to capture data. Normally, my tshark can run in 12 days but yesterday, it suddenly crashed after 2 days of running. On the screen displayed, I didn't see any error. Here is my command:

nohup tshark -i 5 -P -w /tmp/Log.pcap -b filesize:655350|split -b 655350000 - /tmp/log/log- &

About the cronjob, I use 2 cronjob to delete old files:

58 23 * * * rm -rf /tmp/*.bak
59 23 * * * find /tmp/log/* -mtime +0 -exec rm -rf {} \

I'm pretty sure that is is not out of memory or disk space because if so, it was written in the log. No core dump or segmentation fault. My output log is:

640.012974 4240 -> 4593 Camel-v2 298 SACK invoke initialDP
640.013116 4241 -> 4590 Camel 750 SACK invoke initialDP invoke eventReportBCSM invoke applyChargingReport invoke applyChargingReport invoke even

So, nothing wrong except the message in the /var/log

Dec 16 09:16:29 primary kernel: device em3 entered promiscuous mode
Dec 16 09:20:49 primary kernel: device em3 left promiscuous mode
Dec 16 09:31:59 primary kernel: device em3 entered promiscuous mode
Dec 16 09:32:03 primary kernel: device em3 left promiscuous mode
Dec 16 17:09:34 primary kernel: device em3 entered promiscuous mode
Dec 16 17:10:15 primary kernel: device em3 left promiscuous mode
Dec 16 17:11:29 primary kernel: device em3 entered promiscuous mode
Dec 16 17:12:24 primary kernel: device em3 left promiscuous mode
Dec 16 17:13:33 primary kernel: device em3 entered promiscuous mode
Dec 16 17:13:35 primary kernel: device em3 left promiscuous mode
Dec 16 17:14:39 primary kernel: device em3 entered promiscuous mode
Dec 16 17:14:40 primary kernel: device em3 left promiscuous mode
Dec 16 17:15:30 primary kernel: device em3 entered promiscuous mode
Dec 17 19:03:12 primary kernel: device em3 left promiscuous mode
Dec 18 09:03:56 primary kernel: device lo entered promiscuous mode
Dec 18 09:12:41 primary kernel: device em3 entered promiscuous mode

Dec 17 19:03:12 it crashed and never entered back to this mode until I run tshark again on Dec 18 09:12:41. This is exact time my tshark crashed and em3 is my card where I capture data from. I don't know what it means and how to solve this problem. Please help me if you have any idea about this. Thank you so much.

Answer 1

2

As you've been informed many times, tshark (and Wireshark) are liable to run out of memory when run continuously. Until you can come with a convincing explanation as to why this isn't the case with your current issue, most folks are going to assume it's yet another out of memory report.

answered 18 Dec '13, 02:35

grahamb ♦
19.8k●3●30●206
accept rate: 22%

Hi Grahamb, I'm pretty sure that is not the problem of memory because of 2 reasons:

During 3 months, i was running and monitoring tshark very carefully. With the same system, same command, no one else use this server, I realize that tshark is able to run continuously in at lease 12 days. I was checking the statement everyday during 3 months (both of Memory and Disk space) and it has never stopped before 12 days. On the day before it stopped, the memory was still 12%.
If it is the problem of memory, as your warning, it would be informed in the log file before stopping itself, displayed in output of nohup command but as can be seen, no error mentioned in the log except /var/log/message. I also did check the reboot log but nothing wrong, no one rebooted the server, the electricity was fine.

Of course, these reasons are not perfectly to be sure that it is not a problem of memory but as my experience, it quite convinces me. Please feel free to contribute your comment or any idea. ThanksHi Grahamb, I'm pretty sure that is not the problem of memory because of 2 reasons:

(18 Dec '13, 03:22) hoangsonk49

output of nohup command but as can be seen, no error mentioned in the log

You did not post the output of nohup.out, so we can not see that there are no errors!

(18 Dec '13, 04:54) Kurt Knochner ♦

The length of time that tshark can run is related to the traffic captured, not the elapsed run time. Are you 100% certain that during this run the traffic being captured did not change from your previous runs?

The traffic may also have triggered a dissector bug, did you retain the capture file from when it crashed, and if so can it be read by tshark?

(18 Dec '13, 06:18) grahamb ♦

You did not post the output of nohup.out, so we can not see that there are no errors!

I did post. It is output log and I did check but there is no error or core dump or segmentation fault in the whole file. Because it is too large, so I show only some last lines before it stopped:

640.012974 4240 -> 4593 Camel-v2 298 SACK invoke initialDP
640.013116 4241 -> 4590 Camel 750 SACK invoke initialDP invoke eventReportBCSM invoke applyChargingReport invoke applyChargingReport invoke even

These are last lines in the output log of nohup command

Are you 100% certain that during this run the traffic being captured did not change from your previous runs

I fully understand that the running of tshark is related to the traffic, so I checked with some engineers from the Telecom Operator, they confirmed that at that time, there was no error with the network. Also, he showed me the statistic of traffic on that day but it was normal, similar to other days.

The traffic may also have triggered a dissector bug, did you retain the capture file from when it crashed, and if so can it be read by tshark

After it stopped, I collected all the related logs and also the .pcap. I use Wireshark on my PC to read .pcap file, compare some last information in .pcap with output of nohup, with my log written by code but found nothing. All information is identical.

I ask this question because I think maybe the message "device em3 left promiscuous mode" (/var/log/message) could be the clue to find out the reason, and you might have experience with this message but after searching and reading some articles, I understand that this message is just a result, not a reason or clue. So I have nothing else to analyze to find the root cause. I will check the code and print more information to the log.

(18 Dec '13, 17:37) hoangsonk49