This is a static archive of our old Q&A Site. Please post any new questions and answers at

How can I decrypt IKEv1 and/or ESP packets ?


I am using the latest development release. When I try to create a new SA for ISAKMP, it asks for Initiator Cookie and Encryption Cookie. I know the initiator cookie but I am not sure where I can I get this encryption. I know all the configuration of my VPN (encryption algorithm, authentication algorithm, pre-shared key) let me know if it is required here.

I have to tried to input the pre-shared-key there but it does not take it.

Please help. There is no documentation available for this.

asked 18 Jun '12, 06:27

chetan1989's gravatar image

accept rate: 0%

edited 11 Jul '13, 16:51

Kurt%20Knochner's gravatar image

Kurt Knochner ♦

4 Answers:


IKEv1 Decryption

First of all: Wireshark 1.8.0 implements only 3DES and DES for IKEv1 decryption (same for version 1.6.8).

See: epan\dissectors\packet-isakmp.c: decrypt_payload()

If you want do decrypt any other algorithm, the dissector needs to be extended (Volunteers are welcome!). You can file an enhancement request for this at, possibly with a link to this question.

To get the required IKEv1 parameters for the dissector (Initiator's COOKIE and Encryption Key) you need debug output from your IPSEC implementation.

I tested with strongSwan 4.4 on Linux and with this capture file (with the capture file and the data provided in this answer, you can try it yourself). To get the value of "enc key" in the log, you need at least this debug option: --debug-crypt.

Look for ICOOKIE and enc key in the Pluto debug log.

gw205:/# ps auxww | grep pluto
root     24522  0.0  0.3  12572  3488 ?        Ss   15:46   0:00 /usr/libexec/ipsec/pluto --nofork --debug-raw --debug-crypt --debug-parsing --debug-emitting --debug-control --nocrsend --nat_traversal --keep_alive 60

strongSwan ipsec debug log:

2012:07:23-16:40:04 gw205 pluto[24522]: | 2012:07:23-16:40:04 gw205 pluto[24522]: | *received whack message 2012:07:23-16:40:04 gw205 pluto[24522]: | creating state object #12 at 0x9fd77a8 2012:07:23-16:40:04 gw205 pluto[24522]: | ICOOKIE: c6 d1 45 92 85 15 0c 7e 2012:07:23-16:40:04 gw205 pluto[24522]: | RCOOKIE: 00 00 00 00 00 00 00 00 2012:07:23-16:40:04 gw205 pluto[24522]: | peer: c0 a8 8c c8 2012:07:23-16:40:04 gw205 pluto[24522]: | state hash entry 22

2012:07:23-16:40:14 gw205 pluto[24522]: | Skeyid_e: b0 16 81 21 5f 16 20 23 03 18 6d 28 14 dc 56 86 2012:07:23-16:40:14 gw205 pluto[24522]: | ca 5a 47 33 2012:07:23-16:40:14 gw205 pluto[24522]: | enc key: 44 9e 82 9e a9 66 d4 21 fb cb 86 bd 7a d9 2e 86 2012:07:23-16:40:14 gw205 pluto[24522]: | 5a ba b1 5b aa 5c 67 2a 2012:07:23-16:40:14 gw205 pluto[24522]: | IV: dc f8 5e 03 f2 76 ab b9 89 e6 ae ff 46 a9 58 16 2012:07:23-16:40:14 gw205 pluto[24522]: | f4 96 86 25

HINT: If you use any other IPSEC implementation please read the manual how to get that information.

Extract the values of ICOOKIE and ‘enc key’ WITHOUT spaces. HINT: The “enc key” spans two lines!!

ICOOKIE: c6d1459285150c7e
Enc Key: 449e829ea966d421fbcb86bd7ad92e865abab15baa5c672a

Use those values for

Edit -> Preferences -> Protocols -> ISAKMP -> IKEv1 Decryption Table:

Test File: ipsec.pcap

Result without decryption:

IKEv1 main mode - no decryption

IKEv1 quick mode - no decryption

Result with decryption:

IKEv1 main mode - WITH decryption

IKEv1 quick mode - WITH decryption

ESP Decryption

To decrypt ESP packets with Wireshark 1.8.0, you need again debug output from your IPSEC implementation. For Linux and strongSwan, you'll get that information with this command:

ip xfrm state


gw205:/ # ip xfrm state
src dst
        proto esp spi 0x0879355b reqid 16421 mode tunnel
        replay-window 32 flag noecn nopmtudisc af-unspec
        auth hmac(sha1) 0xb8dd42a1c505bed19c2bf23cef00e5d8223c2a5b
        enc cbc(des3_ede) 0xae76ea430b10c72c882c4aeab2283444c54f913d87f5e109
src dst
        proto esp spi 0x1c0d7b38 reqid 16421 mode tunnel
        replay-window 32 flag noecn nopmtudisc af-unspec
        auth hmac(sha1) 0xc364660133b04a4f20e52000dbe4a6ba154c09c1
        enc cbc(des3_ede) 0x39e87c9ca500616b36f2f0d3c7fb688621d7bbf31414abbd

Use those values for the ESP dissector parameters, as shown in the following screenshots. HINT: Take care not to add a space at the end of any parameter (SPI, key, etc.) as decryption will not work in that case.

First enable ESP decryption.

Edit -> Preferences -> Protocols -> ESP -> Attempt to detect/decode encrypted ESP payloads

ESP Parameter

Then add the two ESP SAs (one for each direction!)




If the parameters match the capture file data, Wireshark will be able to dissect the ESP packets.

Result without decryption:

ESP no decryption

Result WITH decryption:

ESP decrypted


answered 23 Jul '12, 09:10

Kurt%20Knochner's gravatar image

Kurt Knochner ♦
accept rate: 15%

edited 17 May '15, 06:37

can you give command: ip xfrm state --- for windows?

(31 Oct '16, 02:05) friis

Could you fix/update the images referenced above? They all show as broken links now.

(26 Jun '18, 10:17) slm


did you add the PSK under the ESP options?

Edit -> Preferences -> ESP -> ESP SAs -> New -> Encryption Key

BTW: Did you check this?

Especially, if your version of Wireshark is built with libcrypt!


answered 18 Jun '12, 07:36

Kurt%20Knochner's gravatar image

Kurt Knochner ♦
accept rate: 15%

edited 18 Jun '12, 07:36

Thanks for your reply.

I don't see option to create New Encryption Key, there is an option to create a new SA (Security Association) wherein I provide the Encryption Key. But when I try to put the same key in ISAKMP Encryption Key field it gives me an error "error in field 'Encryption Key': Error parsing hex string"

I don't know how to get the hex value.

P.S. My wireshark is built with Gcrypt. I don't think that could be a problem because I can decrypt ESP packets easily

(18 Jun '12, 07:46) chetan1989

because I can decrypt ESP packets easily

O.K. I think I misunderstood your request. Do you want to decrypt IKE Phase I packets 5+6 (the encrypted ones) and possibly the whole IKE Phase II traffic?

(18 Jun '12, 08:36) Kurt Knochner ♦

Yes you are correct

(18 Jun '12, 12:08) chetan1989



If you are using racoon (from ipsec-tools), you can see the encryption key from the debug logs of racoon. e.g:

2012-07-11 11:35:55: DEBUG: final encryption key computed: 2012-07-11 11:35:55: DEBUG: 79d5eabc 78ae740b 47258300 f8de371e a4a9da87 4facf41

Also, I had found issues in decryption when i use aes algorithm. With 3des, decryption works fine.

answered 11 Jul '12, 01:53

divya's gravatar image

accept rate: 0%


If you are looking for the cookie and encryption key in vpnc's output, run as vpnc --debug 99 and look for lines matching the following:

WARNING! active debug level is >= 99, output includes username and password (hex encoded)
S4.1 create_nonce
 [2017-02-11 20:15:04]
   i_cookie: da849309 7bd61433
S4.4 AM_packet2
   a635f195 bd412619 17821107 c7d32726 9f5e4781 2ffd7992

answered 11 Feb '17, 20:41

dlenski's gravatar image

accept rate: 0%

edited 11 Feb '17, 20:42