In my heuristic dissector for a private network protocol, I have the following generated treeitem added for 3-way handshaking purposes: The field and value can be seen in the Wireshark packet details window, surrounded by [], and can be used as display filter, but cannot be extracted by MATE for a pdu: Please kindly advise if this is expected, and if there is a workaround – to extract generated protocol treeitem field value. Many thanks in advance. asked 02 Aug ‘17, 01:17  juandering edited 02 Aug ‘17, 01:22  sindy showing 5 of 11 show 6 more comments |
One Answer:
Extracting generated protocol fields (those which are not physically present in frame data) is impossible with the current implementation of MATE which checks that the fields extracted are located within the payload of the indicated transport. Thanks to matey @sindy for the kind guidance in checking this issue. I will now try using associative tables in my dissector to track conversation handshakes. answered 05 Aug '17, 03:58 juandering edited 05 Aug '17, 04:12 sindy |
Just spitballing here: What happens if you leave out the set_generated() call?
Thanks Jaap.
If I leave out the set_generated() call, the result is the same.
Seems MATE extracts field values only if they come with the packet. Generated field values (not from TvbRange) are discarded, or not seen?
If I can make this work with generated field from hash table, then a 3-way handshake like below is peanuts to track for MATE.
First PDU:
SrcMAC to DstMAC \ clud_as_cr with src_ref_id (0x0001)
2nd PDU:
Switched DstMAC to SrcMAC \ clud_as_ack1 with dst_ref_id (0x0001) and src_ref_id (0x3456)
3rd PDU:
SrcMAC to DstMAC \ clud_as_ack2 with dst_ref_id (0x3456) and generated src_ref_id (0x0001)
MATE Gop can have key of SrcMAC, DstMAC, src_ref_id. And this will work even if there are overlaps of the 3-way handshakes.
It is known that MATE has issues even with some “native” dissectors. As it searches for fields only within specified protocol layers, it wouldn’t surprise me if it would ignore generated fields which cannot be physically found in the packet data.
Don’t hesitate to paste the debug output here once you get it.
You were right Sindy, my generated field cannot be discovered / extracted by MATE since it is not really physically included in the packet. In my case, it is just a reference ID from an earlier packet, passed via hash table.
In my 3-way handshake, the original transaction between data source and the destination is many many packets past.
Currently, every time the data sender sends data, it sends a source ref ID. The data receiver acknowledges with Ack1, containing the sender’s source ref ID, and the receiver’s destination ref ID. The data sender sends Ack2 with the data receiver’s destination ref ID.
3-way handshakes can sometimes overlap, and sometimes the sender-destination can also reverse in the “overlap”. So, the only way to track a conversation I see, is to tie the two directed (source and destination) ref IDs with the sender-destination combination.
Unluckily, MATE will not extract the generated source ref ID for Ack2.
Perhaps somebody around has a workaround to share please?
Thanks.
The pdu debug output for the 3 packets - data send with src ref ID, Ack1 with dst and src ref IDs, Ack2 with dst ref ID:
The “found field 0-0” is the result of referencing the Ack2 src ref ID generated field; no TvB range address.
Leaving MATE aside, I’m a bit lost in your explanation, are the refIDs unique per transaction or only unique per source? I do understand that they differ between source and destination and that they are not common between Req and ACK2 of the same trasaction.
Second, MATE was intended as a generic tool allowing to match together packets from different protocols. You are creating your own dissector for a single protocol, so it is far simpler to track the transactions in its code. Take the TCP dissector for loose inspiration, it generates fields called Stream index (
tcp.stream). You can assign arbitrary numbers to your transactions and store them in an associative array indexed by frame number. If, for a given frame number, the transaction number is already assigned, leave it unchanged. If it is not (which indicates the first dissection pass) and you process a request, you assign the next free one and store this newly assigned transaction number to another associative array indexed by the source refID. If handling ACK1, you also check whether a transaction number has already been assigned for that frame number (if not, it indicates the first dissection pass), and if not, you use source refID to find it in the second array; if you don’t find it there, you assign the next free one because it means that the request is missing in the capture. In both cases, you then store the transaction number to a third array, indexed by destination refID. And for ACK2 you act similarly, except that you only search for transaction number in the third associative array.The second and third arrays may have to be indexed by combination of refID and source L3 address if refID alone is not “unique enough”.
There is a Question dealing with similar topic somewhere here but I cannot remember any keyword to identify it.
The ref IDs are unique per source.
My newbie heuristic dissector in Lua, dissects a few other protocols in the company’s suite of network protocols for different applications.
I have not gone yet into conversation or stream tracking yet – and don’t know if Lua is enough, or if we need to go C++.
With Lua, I was hoping to make use of MATE for grouping the related pdus, plus using MATEs timing capabilities for handshake timing validations.
Just now I was trying your suggested associative arrays to track frame numbers already processed, and using L3 Src and Dst MAC as conversation stream keys for the second and third tables to pass the required ref IDs. I think with Lua only, my logic below does not work properly:
if (frames_done[pinfo.number] == nil) then
– process this stream ref1_table[L3SrcMAC .. L3DstMAC] = ref1_id stream_num = stream_num + 1 frames_done[pinfo.number] = stream_num end
If I remove the if testing, the stream_num keeps jumping when I click on a packet in the packet display.
I think this is similar to pinfo.visited issue that from the forum entries I have seen before, does not work with Lua dissectors – needing C++ to work.
Please bear with my newbieness on this matter; plus lack of C++ resources.
Thanks.
1) is confirmed to be impossible with current implementation of MATE, which is the answer to your original Question
2) seems to me to be the same case as the transaction number is a generated field as well, so you can’t use it to build a GoP. So you’ll have to introduce generated fields like
ack1_delay,ack2_delayto deal with the timing in the dissector itself as well.Having said that, I think that given the purpose and rules of this site, it is time that you ask a new Question about your issue which won’t mention MATE, Answer the current one with the conclusion we’ve come to, and accept that Answer.
Well, I think the opposite. Your
frames_donetable is, among other purposes, a replacement ofpinfo.visitedwhich you create and maintain yourself, so any eventual bug renderingpinfo.visitedunusable in Lua doesn’t affect it. I would rather suspect something like declaration of the table in a wrong place, causing it to be re-created with each new call ofyourproto.dissector.I’m in the same boat. An iteration of Lua development is so quick as compared to that of C++ one that I never considered rolling out the full development environment seriously. You’ve just confused me into thinking that you use C by calling an associative array a hash table.
Sindy, I am having difficulties posting the answer. Keep getting red prompt text “* This field is required.” when I click “Anwer Your Own Question”.
Something about my comments seem to be spam? Or too many attempts in posting a comment / answer?
No idea. The site engine started behaving funny recently, and as it is not maintained any more, it is unlikely to bet better. At least it permitted me to convert your Comment into an Answer and slightly edit it (it is not important whether the field is generated by Lua or C++ dissector), now it is your job to Accept it to highlight the Question as usefully answered in the question list for others.