I'm trying to capture a SSLv3 encrypted connection between a target device I'm developing using a PIC32 micro and the Microchip TCP/IP stack (v5.41) + Encryption(v2.6) using Wireshark(v1.6.8). I'm connecting to an openSSL loopback server setup for development testing. When I capture the trace, Wireshark displays only the outgoing packets from my device to the server and does not display the packets from the server to my device. This condition also occurs when I try to connect to the same server using a PC with OpenSSL installed. I'm using a Cisco 2940 switch with one port setup as a SPAN port to monitor the target port and can see normal packet tracing for other operations. Test: What I see in Wireshark with filter: ip.addr == 184.106.243.199 and decoding packets as SSL is the following:
To verify I'm not crazy, I also tried connecting to the google secure server on port 443 as below. What I see in Wireshark with filter: ip.addr == 184.106.243.199 and decoding packets as SSL is the following:
Does anyone have any thoughts on why Wireshark does not seem to display the packets coming from a server on port 8100? Thanks, Bill asked 01 Jun '12, 17:25 William Powell edited 08 Jun '12, 14:27 showing 5 of 6 show 1 more comments |
3 Answers:
Please check if the mirror port on your switch is configured for both directions (ingress and outgress). Please run this command and post the result here:
and/or
where You should NOT see any of these strings in the output (except it is intended ;-)): RX Only or TX only. BTW: What do you see if you change the display filter from ip.addr to:
Regards answered 04 Jun '12, 07:31 Kurt Knochner ♦ edited 04 Jun '12, 08:02 Here's the show monitor output: sh monSession 1Type : Local Session (04 Jun '12, 08:41) William Powell why is dot1q encapsulation enabled? (04 Jun '12, 08:55) Kurt Knochner ♦ When I set the filter to tcp.port eq 8100, I see only the outgoing packets from the client PC, but the SSL connection is made, indicating incoming packets were received. (04 Jun '12, 10:41) William Powell We enabled dot1q encapsulation to see if it made a difference and forgot to remove it. (04 Jun '12, 10:42) William Powell can you please remove dot1q encapsulation and then retry. According to the information you gave, there is (yet) no plausible reason why you can't see the answer packets. May I ask for some further information, just to double check:
What do you see in the capture file now? (04 Jun '12, 15:21) Kurt Knochner ♦
(04 Jun '12, 16:56) William Powell I apologize for the poor formatting quality of these posting. I can't seem to figure out how to do the markdown... Route print:Interface List (04 Jun '12, 17:07) William Powell At one point, I believe the client PC had a MAC miniport driver (e.g. the -Teefer Miniport in the listing above, but I've checked and there is only one ethernet card in the PC. (04 Jun '12, 17:09) William Powell IP config all: Windows IP Configuration
Ethernet adapter Local Area Connection:
(04 Jun '12, 17:10) William Powell Capture file still shows only outgoing packets and bidirectional Ping. How do I upload the output.cap file? (04 Jun '12, 17:27) William Powell Here's the capture file as a hexadecimal text output. I tested by using File->Import using hex option, and it gave me the same display as if I opened the output.cap file:
0000 00 25 b4 37 58 44 00 0d 56 ed 5c d4 08 00 45 00 .%.7XD..V....E. 0010 00 30 43 66 40 00 80 06 22 12 c0 a8 28 75 b8 6a [email protected]"...(u.j 0020 f3 c7 05 b9 1f a4 a1 37 12 28 00 00 00 00 70 02 .......7.(....p. 0030 ff ff 15 13 00 00 02 04 05 b4 01 01 04 02 .............. 0000 00 25 b4 37 58 44 00 0d 56 ed 5c d4 08 00 45 00 .%.7XD..V....E. 0010 00 28 43 67 40 00 80 06 22 19 c0 a8 28 75 b8 6a .([email protected]"...(u.j 0020 f3 c7 05 b9 1f a4 a1 37 12 29 40 d6 df ab 50 10 .......7.)@...P. 0030 ff ff 21 45 00 00 00 00 00 00 00 00 ..!E........ 0000 00 25 b4 37 58 44 00 0d 56 ed 5c d4 08 00 45 00 .%.7XD..V....E. 0010 01 0b 43 68 40 00 80 06 21 35 c0 a8 28 75 b8 6a [email protected]!5..(u.j 0020 f3 c7 05 b9 1f a4 a1 37 12 29 40 d6 df ab 50 18 .......7.)@...P. 0030 ff ff b8 93 00 00 16 03 01 00 de 01 00 00 da 03 ................ 0040 01 4f cd 50 5c a3 a7 4a 28 5c 12 3b f0 14 22 f9 .O.P..J(.;..". 0050 48 8d 30 cb 09 40 df 1c d1 62 48 c9 54 da 60 a4 [email protected] 0000 00 25 b4 37 58 44 00 0d 56 ed 5c d4 08 00 45 00 .%.7XD..V....E. 0010 00 63 43 6d 40 00 80 06 21 d8 c0 a8 28 75 b8 6a [email protected]!...(u.j 0020 f3 c7 05 b9 1f a4 a1 37 14 17 40 d6 e3 26 50 18 [email protected]&P. 0030 fc 84 36 23 00 00 14 03 01 00 01 01 16 03 01 00 ..6#............ 0040 30 bd 04 9b cd 2a 6b 4e 4b 71 cf 78 ca 82 ab 6d 0....*kNKq.x...m 0050 0c 77 18 8d 97 a9 f2 7c be 33 5e 02 23 04 d9 e5 .w.....|.3^.#... 0060 17 9f 2c 80 f8 da 8c c4 7f 9e 9e 4c a1 74 21 d4 ..,........L.t!. 0070 4a J 0000 00 25 b4 37 58 44 00 0d 56 ed 5c d4 08 00 45 00 .%.7XD..V....E. 0010 00 28 43 6e 40 00 80 06 22 12 c0 a8 28 75 b8 6a .([email protected]"...(u.j 0020 f3 c7 05 b9 1f a4 a1 37 14 52 40 d6 e3 61 50 10 [email protected] 0030 fc 49 1f 1c 00 00 00 00 00 00 00 00 .I.......... 0000 00 25 b4 37 58 44 00 0d 56 ed 5c d4 08 00 45 00 .%.7XD..V....E. 0010 00 28 43 6f 40 00 80 06 22 11 c0 a8 28 75 b8 6a .([email protected]"...(u.j 0020 f3 c7 05 b9 1f a4 a1 37 14 52 40 d6 e3 61 50 14 [email protected] 0030 00 00 1b 62 00 00 00 00 00 00 00 00 ...b........ 0000 00 25 b4 37 58 44 00 0d 56 ed 5c d4 08 00 45 00 .%.7XD..V....E. 0010 00 3c 43 7c 00 00 80 01 61 f5 c0 a8 28 75 b8 6a .<C|....a...(u.j 0020 f3 c7 08 00 34 5c 02 00 17 00 61 62 63 64 65 66 ....4....abcdef 0030 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76 ghijklmnopqrstuv 0040 77 61 62 63 64 65 66 67 68 69 wabcdefghi 0000 00 0d 56 ed 5c d4 00 25 b4 37 58 44 08 00 45 00 ..V...%.7XD..E. 0010 00 3c 38 a1 00 00 73 01 79 d0 b8 6a f3 c7 c0 a8 .<8...s.y..j.... 0020 28 75 00 00 3c 5c 02 00 17 00 61 62 63 64 65 66 (u..<....abcdef 0030 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76 ghijklmnopqrstuv 0040 77 61 62 63 64 65 66 67 68 69 wabcdefghi 0000 00 25 b4 37 58 44 00 0d 56 ed 5c d4 08 00 45 00 .%.7XD..V....E. 0010 00 3c 43 7d 00 00 80 01 61 f4 c0 a8 28 75 b8 6a .<C}....a...(u.j 0020 f3 c7 08 00 33 5c 02 00 18 00 61 62 63 64 65 66 ....3....abcdef 0030 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76 ghijklmnopqrstuv 0040 77 61 62 63 64 65 66 67 68 69 wabcdefghi 0000 00 0d 56 ed 5c d4 00 25 b4 37 58 44 08 00 45 00 ..V...%.7XD..E. 0010 00 3c 38 d9 00 00 73 01 79 98 b8 6a f3 c7 c0 a8 .<8...s.y..j.... 0020 28 75 00 00 3b 5c 02 00 18 00 61 62 63 64 65 66 (u..;....abcdef 0030 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76 ghijklmnopqrstuv 0040 77 61 62 63 64 65 66 67 68 69 wabcdefghi 0000 00 25 b4 37 58 44 00 0d 56 ed 5c d4 08 00 45 00 .%.7XD..V....E. 0010 00 3c 43 7e 00 00 80 01 61 f3 c0 a8 28 75 b8 6a .<C~....a...(u.j 0020 f3 c7 08 00 32 5c 02 00 19 00 61 62 63 64 65 66 ....2....abcdef 0030 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76 ghijklmnopqrstuv 0040 77 61 62 63 64 65 66 67 68 69 wabcdefghi 0000 00 0d 56 ed 5c d4 00 25 b4 37 58 44 08 00 45 00 ..V...%.7XD..E. 0010 00 3c 38 f4 00 00 73 01 79 7d b8 6a f3 c7 c0 a8 .<8...s.y}.j.... 0020 28 75 00 00 3a 5c 02 00 19 00 61 62 63 64 65 66 (u..:....abcdef 0030 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76 ghijklmnopqrstuv 0040 77 61 62 63 64 65 66 67 68 69 wabcdefghi 0000 00 25 b4 37 58 44 00 0d 56 ed 5c d4 08 00 45 00 .%.7XD..V....E. 0010 00 3c 43 7f 00 00 80 01 61 f2 c0 a8 28 75 b8 6a .<C.....a...(u.j 0020 f3 c7 08 00 31 5c 02 00 1a 00 61 62 63 64 65 66 ....1....abcdef 0030 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76 ghijklmnopqrstuv 0040 77 61 62 63 64 65 66 67 68 69 wabcdefghi 0000 00 0d 56 ed 5c d4 00 25 b4 37 58 44 08 00 45 00 ..V...%.7XD..E. 0010 00 3c 39 51 00 00 73 01 79 20 b8 6a f3 c7 c0 a8 .<9Q..s.y .j.... 0020 28 75 00 00 39 5c 02 00 1a 00 61 62 63 64 65 66 (u..9....abcdef 0030 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76 ghijklmnopqrstuv 0040 77 61 62 63 64 65 66 67 68 69 wabcdefghi <\code> (04 Jun '12, 17:44) William Powell that looks all pretty much O.K. I don't see a plausible reason why it does not work. Now, there are only a few options left.
(04 Jun '12, 21:11) Kurt Knochner ♦ Kurt, Thanks for all of your help with troubleshooting this problem. I'll try your suggestions above, but I'm not hopeful. (05 Jun '12, 08:39) William Powell Do you have openssl installed on your PC? If you wanted to try the test for yourself, we got our opensll installation from the following site: http://slproweb.com/products/Win32OpenSSL.html Thanks again for your help. (05 Jun '12, 08:41) William Powell Does anyone have openssl installed and is willing to try the openssl tests and wireshark capture indicated in the original question above? (05 Jun '12, 08:42) William Powell 1 Done. It works as expected. There is no reason why it should not work. After all, you do see one half of the whole TCP connection in Wireshark. Again, there must be a yet unknown problem with your sniffer setup. Either it's the switch or it's the sniffer PC. Try sniffing on the client PC, it nothing else works. Very strange effect... I'm sorry, but I'm running out of ideas. Maybe someone else here wants to pick up? (05 Jun '12, 10:09) Kurt Knochner ♦ Thanks for trying the test and confirming that something is going on on my network or my sniffer PC. (05 Jun '12, 12:12) William Powell O.K. final option: Get a live image of BackTrack 5R2 (contains Wireshark) and run it on your sniffer PC.
You can also try YUMI. It's a very nice tool to create a bootable flash drive with a ton of different linux systems (including BT5R2) on a stick.
Boot your sniffer PC from the stick and use Wireshark. If that does not eliminate the problem, there is either something wrong with your switch, OR somebody sold you a filtering patch cable ;-) (05 Jun '12, 12:25) Kurt Knochner ♦ showing 5 of 18 show 13 more comments |
What kind of software is running on the sniffer PC? There are quite a few issues with interfering software, of of which might cause this effect. It is best to use a system as clean as possible to do the capturing. Running a LiveCD/USB-stick with a linux distribution (as Kurt suggested) is a good way to find out with your current Sniffer PC if it is indeed receiving all data from the Span port. answered 05 Jun '12, 15:33 SYN-bit ♦♦ Oh BTW, I also did run a test with OpenSSL against the SSL loopback server and saw traffic in both directions (captured on my client, not on a span-port):
(05 Jun '12, 15:35) SYN-bit ♦♦ 1 Finally... After Kurt and SYN-bits suggestion to get a clean PC and Kurt's ability to do the openssl test successfully, I increased my focus on the sniffer PC and potential interfering software. On the sniffing PC, Symantec Endpoint Protection is installed. I talked to our IT group and we examined the endpoint protection rules but did not find anything specifically related to port 8100. However, I found a computer that I could temporarily disable the Symantec "Network Threat Protection" and I'm now able to see outgoing AND incoming packets on port 8100. Looks like I'll need to ask our IT group to provide a rule for the Endpoint Protection software to allow this port. Thanks to everyone for their help to investigate this problem and finding a solution. (06 Jun '12, 10:26) William Powell Thanks for all the answers and good suggestions. They were helpful in tracking down the problem. Sorry to not have posted this answer sooner. After SYN-bit proved that openSSL worked with the server, I continued to investigate with our IT department and found that the Symantec Endpoint protection (on the Wireshark PC) had a filter that was denying incoming access to port 8100 (and other ports) for traffic not originating from the PC to that IP/port resulting in Wireshark not being able to trace the returning packets. (25 Sep '12, 13:35) William Powell |
McAfee Users: In McAfee there is a binding within the Interface settings that needs to be unchecked to allow for full TX & RX capture: McAfee NDIS Intermediate Filter Miniport Regards. answered 25 Sep '12, 11:55 Jamie_Atkins Trend Micro Deep Secure users: Uncheck "Trend Micro LightWeight Filter Driver" at the adapter. (09 Apr '14, 03:31) kaj |
1.I did not use any capture filter (unless there is a default that is being used by Wireshark), just a display filter. Is there a way for me to verify no capture filter is being used?
2.I'm not sure of the network setup after the Cisco 2940 switch. I believe it is connected to another Cisco switch that is using VLANs and then ... eventually gets to the internet.
3.There is a single Ethernet interface on the client PC
4.Tried a ping this morning and I can see both outgoing and incoming ICMP messages.
There is no default filter. Please check this: Capture -> Options -> Capture Filter (looks totally different in 1.7.x).
Just to be sure. You do see both ICMP messages in wireshark (attached to a mirror port on a switch), but you don't see a TCP reply (SYN-ACK and others) with the same wireshark setup to the same endpoint? HOWEVER, you do see bidirectional traffic from the same client (with the same wireshark setup) to google?
Opened the capture options dialog and I don't see anything specified.
On your last question:
- I do see both incoming and outgoing ICMP per the trace above,
- I do NOT see both incoming and outgoing TCP to the same endpoint using the ssl test command.
- I do see bidirectional traffic from the same client and wireshart setup when connected to google using the ssl test command.
Here is the ping trace with display filter ip.addr == 184.106.243.199
160 14.143801 192.168.40.118 184.106.243.199 ICMP 74 Echo (ping) request id=0x0200, seq=59650/745, ttl=128
161 14.154775 184.106.243.199 192.168.40.118 ICMP 74 Echo (ping) reply id=0x0200, seq=59650/745, ttl=115
184 15.129103 192.168.40.118 184.106.243.199 ICMP
One thing that might provide a clue is that port 8100 is listed as a registered port in the IANA list - Xprint Server, which sounds like something Wireshark might want to filter out (e.g. incoming traffic). I'm not sure why the test server is on this port.