This is a static archive of our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

From PCAP to SQL

0

I fill an SQL DB with results from a Lua script launched by tshark like eth.src, dhcp.hostname, etc ... I tried to do all in Lua, but wasn't successful on the SQL part, reinventing the wheel so I used Python to insert/update data into DB.

For the moment, I'm using a CSV file as an intermediary: Lua parses the whole PCAP, creates a CSV file, and a Python script updates CSV to SQL. The whole is managed by a bash script, it's a bit heavy and not really efficient with so many scripts to handle.

Is there a better way to communicated between those two languages like socket / pipe / output? What would be more efficient alternatives to fill such SQL DB from PCAP?

asked 08 Nov '15, 21:43

TomLaBaude's gravatar image

TomLaBaude
66171724
accept rate: 66%


2 Answers:

1

Is there a better way to communicated between those two languages like socket / pipe / output?

Why are you using the Lua script at all? You could run tshark and parse the output with python (which then adds data to the database).

tshark -nr input.pcap -Y "dhcp" -T fields -e eth.src -e eth.dst -e ip.src -ip.dst -e dhcp.hostname -E header=y -E separator=; | python yourscript.py

Regards
Kurt

answered 09 Nov '15, 14:41

Kurt%20Knochner's gravatar image

Kurt Knochner ♦
24.8k1039237
accept rate: 15%

edited 09 Nov '15, 14:41

I use Lua to retrieve hexa values of some filters like "wlan_mgt.ssid" if some non ASCII chars are printed, like explained in https://ask.wireshark.org/questions/43521/retrieve-hex-values-in-lua

Can you get such hexa values with tshark -T fields?

(10 Nov '15, 00:39) TomLaBaude
1

In that case you could use '-T pdml' or '-T psml' instead of '-T fields' and parse the XML like structure, which 'should' contain hex values as well. If that does not work, you can still go the 'brute force' route with

tshark -nr input.pcap -Vx | python script.py

And if that does not contain the values in HEX, your Lua/Python mix is probably the best option already ;-)

Maybe you can drop the bash script and call tshark (with the Lua script parameters) directly from your python script...

Regards
Kurt

(10 Nov '15, 04:48) Kurt Knochner ♦

PyShark could be another option, information regarding it can be found here: http://kiminewt.github.io/pyshark/

(27 Apr '16, 15:24) kim

0

I use tshark to export to csv. The "pandas" library can then load the csv (pandas.read_csv)and export it to sql (pandas.write_sql IIRC ?). This might be more efficient then your current script depending on how you convert things.

answered 28 Apr '16, 14:14

teto's gravatar image

teto
6223
accept rate: 0%