Can't decrypt ssl in capture from windump

How do you configure this? In preferences/protocols/ssl you can browse for a file, but i notice that you can only specify files that already exist. I created a file, and specified it, but the file is always zero length.

The file selection dialog behaves kind of strange sometimes. It's probably better to just paste the full path of the file to the field SSL debug file

if anyone could tell me how to get the ssl debug log going

The only thing you need to do is to specify the debug log. Then close Wireshark, empty or delete the file and restart Wireshark. The file will then be filled with debug messages, if the user that runs Wireshark has enough rights to create the file and to write to it, as Wireshark will run in the user context. If that does not happen on your system, please add the following information

• OS and OS version
• Wireshark version
• full path to the debug file

a lot of the packets are showing red ("Bad Checksum"). Is this a problem?

No. That's most certainly due to checksum offloading, as you were capturing on the server itself.

should I use the ip returned by nslookup, or the ip that that shows up as the destination in the trace?

Of course the IP in the trace file. That's the only one Wireshark can see.

How is the ssl debug log configured? Will the log be written to when wire shark is analyzing a trace file,

Yes. see above.

Then i converted the pem, and placed the rsa private key portion into a separate file.

The key file should look like this:

-----BEGIN RSA PRIVATE KEY-----
MIICXQIBAAKBgQC8/gQ2y24WaJuuoF4cAibeqLOU7JnkeX21ozjVe7tpeT1ZqSMc
gRLUGAGquCmIT7XipvWLfll7itGGiCMTXdAqNPgiWvxdNVhMh8W/vlv7xTdnbquY
... some lines ....
BMb1winyNNcTHu1vLScCQQCrYa+AfS879mK+L3kFXrliXnJ5+4uBsY55dUQc8j97
CKuarIR9Pb4OAE1mAg6SIWoXOaIp0XmFtXtohBmav3ex
-----END RSA PRIVATE KEY-----

If you see the following (or similar) lines in the key file

-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,530703DFD90796F4

kEa20CNx8Pn5zw2sq/UtI5BFk0IlXImzDzdh9Mlv8LdIHwz67rtwQTDstJf9Wucj

then your exported key is encrypted and you must give Wireshark the passphrase for the key.

Edit -> Preferences -> Protocols -> SSL -> RSA Key List --> Password: within the key entry.

If the key is not encrypted and it still does not work, the following 'problem' occurs sometimes:

Your SSL/TLS connection is using a cipher that cannot be decrypted with only the RSA key of the server. That's all ciphers with DH in the name (Diffie Hellman). The ssl debug log will show the cipher used, as well as Wireshark itself, by looking at the SSL handshake.

Add a custom column in the GUI with the following field, to view the negotiated cipher suite.

ssl.handshake.ciphersuite

If the connection is using a DH cipher, you need to SSL session keys, exported by your client. See here:

https://developer.mozilla.org/en-US/docs/NSS_Key_Log_Format https://www.google.com/?q=site%3Aask.wireshark.org+SSLKEYLOGFILE+ http://wiki.wireshark.org/SSL

Read the questions and answers about using the RSA key log file generated by a browser (by using the environment variable SSLKEYLOGFILE).

Finally, try to decrypt a SSL session that is known to work.

http://wiki.wireshark.org/SampleCaptures#SSL_with_decryption_keys

Regards
Kurt

"I have exported the server cert, converted to pem and separated out the private key."

Hm, if that was possible all it would take to decrypt a SSL/TLS session would be to trace the negotiation .
The private key is not sent during negotiation, that is kept at the server. So unless you are given the private key by the server's administrator, there is no way to decrypt the traffic.