which protocol to use for decoding a pcap

(I'm assuming here that the protocol is the lowest-level protocol in the packets, just as, for example, IEEE 802.3 Ethernet would be the lowest-level protocol in an Ethernet capture file. If not, then there is no field in pcap files for higher-level protocols; either the protocol running below your protocol must have a field that specifies the next protocol - for example, the Ethernet type field in Ethernet packets - or it must always have the same protocol as the next protocol or must have some other way to specify the next protocol. In the first case, you'd have to register a dissector table for that field and use that; in the second case, the dissector for your protocol would always call the next protocol's dissector; in the third case, you'd probably need a preference to specify the next protocol.)

The field in question is the network field in the file header in the pcap file format and the LinkType field in an Interface Description Block record in the pcap-NG file format.

The values used in that field are the same in pcap and pcap-NG format. Those values are assigned by the Tcpdump Group by sending a request to [email protected] The current set of values is described by the list of link-layer header types at the Tcpdump Group's Web site.

However, those values are, within Wireshark, only known by the module in the Wiretap library (the library that reads capture files) that is common to pcap and pcap-NG files; that module is in the wiretap/pcap-common.c file. The rest of Wireshark has its own set of packet-type encapsulation values, and maps the link-layer header-type values for various capture file formats into those values. A dissector would register itself for a particular packet-type encapsulation value with a call to dissector_add_uint(), the first argument of which would be "wtap_encap", and the second argument of which would be the Wireshark encapsulation type value (NOT the link-layer type value assigned by the Tcpdump Group!).

You have two choices here:

1. use one of the "reserved" values in the range 147 through 162 - LINKTYPE_USER0 through LINKTYPE_USER15 - and change the preference for the corresponding DLT_ value - DLT_USER0 through DLT_USER15 - in the DLT_USER protocol preference to use your protocol;
2. request a standard LINKTYPE_ value from [email protected] and, once it's assigned, add a WTAP_ENCAP_ value for it (if there isn't already one), and modify the pcap_to_wtap_map[] table in wiretap/pcap-common.c to map the assigned LINKTYPE_ value to the corresponding WTAP_ENCAP_ value.