Greetings, My organization wants to embrace the file format of PCAP-NG as the type of file used throughout business models over a former text-formatted log. So far, only a couple of their tools have already done so, but there are many more that will eventually have to have their implementation converted. Aside from looking at the format standards online, we have a need to verify the format these tools use to generate these PCAP, or PCAPNG (preferably), files once they are converted to generate them. Are there any tools out there that can provide this verification? I've ran into capinfos.exe within Wireshark and, though it gives information about the packet, to include its format, it does not verify it. Or, does it? Thanks in advanced. asked 11 Aug '15, 12:50  coloncm showing 5 of 8 show 3 more comments |
One Answer:
pcap-ng is extensible, so, obviously, only the stuff in the spec can be verified, and a verifier can become out of date if the spec is updated. Given that, the Wireshark code to read various capture file formats does a fair bit of sanity checking, so, whilst it may not verify every single requirement in the pcap-ng spec, it should return errors for, at least, the violations of those requirements for which it can't process the file, so capinfos is probably a reasonably good verifier. Note, though, that it might try to work around some violations, such as the captured length of a packet being larger than the original length. A verifier, explicitly written to check every requirement in the spec, might be a useful program to have. answered 11 Aug '15, 15:07  Guy Harris ♦♦ The fact that Wireshark attempts at working around problems with formats on files is what makes it not necessarily a standard format verifier. As for capinfos.exe, I don't think providing information about a PCAP, or PCAPNG, file and its contents necessarily verifies that it meets format standards. An error would simply means that there is at least one thing wrong with the file that cannot be repaired, or ignored, on-the-fly. Ideally, there ought to exist a tool out there, or a process for that matters, that could provide information on what a given file needs to meet format standards. Else, can a given format really a standard to abide by? Just asking :) (13 Aug '15, 13:35) coloncm Yes, I think somebody even said "A verifier, explicitly written to check every requirement in the spec, might be a useful program to have." on this thread. :-) (13 Aug '15, 14:12) Guy Harris ♦♦ Thanks for clarifying. We're apparently on the same page. So, it appears, given your position and experience, that there doesn't exist such tool since you suggest that I come in to a mailing list and suggest that one ought to exist. I'll do this, but I'm still inclined to believe that someone must've at least developed a process for doing such work, and I'm interested in knowing about it. (14 Aug '15, 09:37) coloncm Nobody's told me about one, and I'm on that list. (14 Aug '15, 10:57) Guy Harris ♦♦ 1 I don't think there is a tool like that. The closest match may be TraceWrangler, which can display the block structure and some of the values you can't see in Wireshark, but that's not really close to a full fledged verification tool. I agree that it may be good to have something like that, but it is a lot of work for sure to create one. https://www.tracewrangler.com/documentation/TraceWrangler.html?ThePCAPngstructureviewer.html (14 Aug '15, 13:12) Jasper ♦♦ Thanks, Jasper. It does at least provide a structure view of the file, and one can examine the structure for human determination. So, it's a great start for me. (18 Aug '15, 09:51) coloncm If you have any trouble with the structure viewer let me know ;-) (18 Aug '15, 16:39) Jasper ♦♦ By complete coincidence, Michal Labedzki recently submitted a PCAPNG file format reader for wireshark that just got into the main code today, so should be in the latest 1.99.9 development builds on the automated builds server. It generates a few expert infos for malformed portions, as well, and generates "Unknown" for proprietary block codes and so on. Someone could probably write a script to invoke tshark and parse the output to see if it's a valid PCAPNG or not. (or a Lua plugin could do it) (18 Aug '15, 19:44) Hadriel If you hand a pcap-ng file to TShark, it'll read the capture, which is what it's supposed to do with pcap-ng files, not parse it as a pcap-ng file dissecting the pcap-ng file format. There isn't yet a "file dissector" program, unfortunately, so the pcap and pcap-ng dissectors aren't actually wired up to anything. (18 Aug '15, 23:36) Guy Harris ♦♦ They are - just in a completely unintuitive/non-obvious way: using the "MIME Files Format" format. So in Wireshark you open the Open Files dialog window (ctrl-o) and select the "MIME Files Format" from the file type drop-down box instead of "Automatic", and it should open a PCAPNG file showing the file format. In tshark you would use the (19 Aug '15, 12:36) Hadriel Thanks, Jasper. I can at least confirm the structure of the file with this viewer; very handy! (27 Aug '15, 06:47) coloncm showing 5 of 11 show 6 more comments |
Note that it's a file format strongly oriented towards time-stamped events, with an emphasis on the events being messages transmitted between entities either on the same machine or connected by some form of communication medium, so it may or may not be appropriate for all uses.
BTW, somebody in your organization, perhaps you, should join the pcap-ng-format mailing list if you're going to standardize on pcap-ng for your purposes. (That might be the best place to, for example, suggest a verifier; it's not a "Wireshark file format", it's a file format that Wireshark might, right now, support better than any other software, but that's subject to change - for example, I plan to add new file-reading and file-writing APIs to libpcap/WinPcap to expose its full capabilities, with those APIs also being able to read pcap files.)
"Note that it's a file format strongly oriented towards time-stamped events, with an emphasis on the events being messages transmitted between entities"
This is precisely why it preferred over standard PCAP format.
"somebody in your organization, perhaps you, should join the pcap-ng-format mailing list if you're going to standardize on pcap-ng for your purposes."
This is a great suggestion for keeping up with its changes.
"That might be the best place to, for example, suggest a verifier"
This post's hope is to obtain knowledge of an existing verifier of standards on either a PCAP or PCAPNG format on files. I don't think we're there, yet, to be able to offer a standard format verifier to anyone.
"it's not a "Wireshark file format", it's a file format that Wireshark might, right now, support better than any other software, but that's subject to change"
I am well aware of it and, in fact, know that a file may or may not be in the correct format just because Wireshark is able to open it (or even create/save it for that matters). Again, it's reason why I posted this question. Thanks for your comments/suggestions
Sorry, what I meant by "suggest a verifier" was "suggest that there should be a verifier", not "suggest a piece of software to use as a verifier".
What I meant was "don't use it as a container for arbitrary data" - pcap format is also "a file format strongly oriented towards time-stamped events, with an emphasis on the events being messages transmitted between entities", it's just more strongly oriented, as, for example, the time stamps are a mandatory part of every record, and there's no provision for records other than packet records.
What I meant was "the format doesn't belong to Wireshark, so further discussion of this issue - or any other issues, such as new block types or options you might want - is best done on the mailing list for pcap-ng, rather than in Q&A sites/mailing lists specific to Wireshark".
Why would you do that? While PCAP-NG is an open format, it has a strong relation to packet capturing. So, why use PCAP-NG for business model data transfers (maybe I'm misunderstanding something here). Wouldn't JSON, XML or something similar be a much better format for your business applications?
I understand your rational completely. Once upon a time, XML was heavily embraced.
As of late, most tools developed by my organization made the transition to PCAP and PCAP-NG, or added this functionality, simply to support the integration of Wireshark because a string of dissectors were created that replicated their functions, streamlined the speed of service extensively, and got management to think about reducing the number supported formats for simplicity and standardization.
Our tool-set is more onsite, closed-network-driven than typical organizations which rely on web-based, internet integration, so I can't really blame them for that.