This is our old Q&A Site. Please post any new questions and answers at

I am trying to decrypt ESP payloads with AES-GCM as the encryption algorithm. I am able to identify the correct settings for all of the fields, but I am not clear on what to use for the encryption key and authentication algorithm, key settings. I have tried various configurations and while I can get decrypted data, it is not dissected and I am guessing that the decryption is not correct.

I used the command ip xfrm state to obtain the SA information. It shows that AES-GCM is used with 256 bit key. Do I use the entire 72 octet keymat supplied in xfrm output for the authentication key?

For the authentication in am using the ANY 128 bit authentication [no checking} - setting. Is this correct?

I am using wireshark 1.10.6

Thx, Rich

asked 02 Apr '14, 06:49

Rich's gravatar image

accept rate: 0%

Did you follow the steps outlined in my answer to the following question?

see ESP decryption in that answer.


permanent link

answered 02 Apr '14, 12:43

Kurt%20Knochner's gravatar image

Kurt Knochner ♦
accept rate: 15%


I appreciate the feedback. I have read through and followed the approach outlined in your reference. I have been looking for a successful example that is specific to the particular algorithm aes-gcm and have yet to see one.

This ESP transform does not have an explicit authentication algorithm. I am not sure how to correctly map the keymat generated by the xfrm command into the key field for the SA ui form. We are using 256 bit key so the keymat provides 32 octets (64 hex digits) and 4 octets used a slat value for the nonce. I can only get an indication of decryption when I plug-in the whole keymat field (key+ nonce).

We also have a 128 bit (16 octet ICV). I am not sure if there needs to be a setting configured for the authentication algorithm and key fields so have been using any 128 bit authentication [no checking] to at least the correct size for the ICV.


(03 Apr '14, 07:23) Rich

Likewise, I came here looking for a way to make Wireshark aware of rfc4106(gcm(aes)) and instead saw a reference to a page that assumes, as Wireshark seems to, that we want to use cbc(aes) (which is for IKEv2, rather than ESP) for ESP using aead.

(01 Mar '16, 10:01) Jerry Miller
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here



Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text]( "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:


question asked: 02 Apr '14, 06:49

question was seen: 4,205 times

last updated: 01 Mar '16, 10:34

p​o​w​e​r​e​d by O​S​Q​A