This is our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

Hi Friends,

This is my first post and will try to make it as informative as I can. The issue is related to remote desktop connection on one of the windows 2008 R2 server. Some users are facing issue while connecting to remote desktop session of a server in VLAN A. For the same server remote desktop for other users are working fine. While troubleshooting Network team was able to see traffic flowing through the firewall and reaching to the server but never responded back. I did setup wireshark packet capture on the server side to analyse the traffic. I do see packets reaching to the server but are not being responded back. There is packet retransmission happening from client side due to no ACK. Can any one suggest what could be wrong here?

alt text

asked 17 Jun '15, 04:39

vjay's gravatar image

vjay
11116
accept rate: 0%


I see three possible reasons:

  • you don't see the server response (SYN-ACK( due to a known "problem" of your capture setup (see here: https://ask.wireshark.org/tags/outgoing/)
  • there is a SYN-ACK, but the server is dual homed (multiple interfaces in the same subnet) and it sends the response out a different interface.
  • there is no SYN-ACK, which means the RDP server did not answer the SYN. That's something you can only diagnose with the help of microsoft.

Personally I believe it's option #2, although this is just a wild guess: I've seen that happen from time to time, because people give their servers several interfaces for "redundancy" reasons and then they simply assign an IP address from the same subnet (Windows does not prevent that)! This usually works on a local network without security devices (firewall, loadbalancers, etc.), but it can cause problems if those devices are in place.

In your case, the firewall might well block the SYN-ACK, if it's coming from a different MAC address than the SYN was sent to (windows sends the SYN-ACK out the other interface). This depends on the firewall type and on the firewall configuration.

Regards
Kurt

permanent link

answered 17 Jun '15, 04:50

Kurt%20Knochner's gravatar image

Kurt Knochner ♦
24.8k1039237
accept rate: 15%

Thanks Kurt for your time. You are spot on. We do have two interfaces on that server over different VLAN but I couldn't capture syn/ack on other interface. I applied same filter as first interface which receives SYN packets. Is there anyway I can capture those SYN/ACK packets on second interface?

(17 Jun '15, 05:18) vjay

Set Wireshark to capture from all relevant interfaces. Simply check all the interfaces you wish to capture from in the Capture Options dialog.

(17 Jun '15, 05:27) grahamb ♦

to rule out problem #1 you should capture on a switch mirror/monitor/span port, which mirrors the traffic of both server interfaces.

(17 Jun '15, 06:19) Kurt Knochner ♦

Well, I am getting all the hand shakes and frames for other communication only for this particular communication I am seeing this behaviour. RDP connections are working for internal traffic but the issue is for external traffic where we are only getting SYN packets in capture. For internal traffic we are getting ACK packets from same interface. Do you think problem#1 and problem#3 can still be an issue?

(17 Jun '15, 06:50) vjay

but the issue is for external traffic where we are only getting SYN packets in capture.

O.K. can you please provide a capture file for the case where it does not work? I want to check if there is something wrong in the frame with the SYN flag.

Do you think problem#1 and problem#3 can still be an issue?

Maybe. Can you asure that the frames in both cases (internal and 'external') are taking the same path/route?

(17 Jun '15, 07:04) Kurt Knochner ♦

Pardon me for ignorance, How can i upload a file here?

(17 Jun '15, 07:29) vjay

You can't. Please upload it to dropbox, google drive or cloudshark.org and then post the link here.

(17 Jun '15, 07:39) Kurt Knochner ♦
(17 Jun '15, 07:47) vjay

The SYN frame looks good, but as you can see there is also no answer to the ping requests.

So, this leads me to further assumptions

  • Do you have the windows firewall enabled on that windows system or any other seecurity software (Endpoint security) that might block those frames?
  • Do you have a default route pointing back to the firewall or at least a route for the remote network?

Can you please post the output of the following command on the Windows server?

route print

(17 Jun '15, 07:59) Kurt Knochner ♦

Pasting it in two parts.

C:\Users>route print
===========================================================================
Interface List
 12...00 0c 29 0c 88 7b ......Intel(R) PRO/1000 MT Network Connection #2
 11...00 0c 29 0c 88 71 ......Intel(R) PRO/1000 MT Network Connection
  1...........................Software Loopback Interface 1
 13...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter
 14...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #2
 15...00 00 00 00 00 00 00 e0 Teredo Tunneling Pseudo-Interface
===========================================================================

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0      172.16.87.5    172.16.87.123    266
      62.116.24.9  255.255.255.255      172.16.84.1    172.16.84.121     11
       80.90.2.33  255.255.255.255      172.16.84.1    172.16.84.121     11
     88.81.157.71  255.255.255.255      172.16.84.1    172.16.84.121     11
    91.220.189.12  255.255.255.255      172.16.84.1    172.16.84.121     11
        127.0.0.0        255.0.0.0         On-link         127.0.0.1    306
        127.0.0.1  255.255.255.255         On-link         127.0.0.1    306
  127.255.255.255  255.255.255.255         On-link         127.0.0.1    306
        172.0.0.0        255.0.0.0      172.16.84.1    172.16.84.121     11
      172.16.84.0    255.255.255.0         On-link     172.16.84.121    266
    172.16.84.121  255.255.255.255         On-link     172.16.84.121    266
    172.16.84.255  255.255.255.255         On-link     172.16.84.121    266
      172.16.87.0    255.255.255.0         On-link     172.16.87.123    266
    172.16.87.123  255.255.255.255         On-link     172.16.87.123    266
    172.16.87.255  255.255.255.255         On-link     172.16.87.123    266
        224.0.0.0        240.0.0.0         On-link         127.0.0.1    306
        224.0.0.0        240.0.0.0         On-link     172.16.87.123    266
        224.0.0.0        240.0.0.0         On-link     172.16.84.121    266
  255.255.255.255  255.255.255.255         On-link         127.0.0.1    306
  255.255.255.255  255.255.255.255         On-link     172.16.87.123    266
  255.255.255.255  255.255.255.255         On-link     172.16.84.121    266
===========================================================================
(17 Jun '15, 08:12) vjay
Persistent Routes:
  Network Address          Netmask  Gateway Address  Metric
        172.0.0.0        255.0.0.0      172.16.84.1       1
     88.81.157.71  255.255.255.255      172.16.84.1       1
       80.90.2.33  255.255.255.255      172.16.84.1       1
      62.116.24.9  255.255.255.255      172.16.84.1       1
    91.220.189.12  255.255.255.255      172.16.84.1       1
          0.0.0.0          0.0.0.0      172.16.87.5  Default
===========================================================================

IPv6 Route Table
===========================================================================
Active Routes:
 If Metric Network Destination      Gateway
  1    306 ::1/128                  On-link
  1    306 ff00::/8                 On-link
===========================================================================
Persistent Routes:
  None

(17 Jun '15, 08:13) vjay

Opps. Let me see any other way I can upload.

(17 Jun '15, 08:20) vjay

You have two gateways. One for 172.0.0.0/8 and one default route.

          0.0.0.0          0.0.0.0      172.16.87.5    172.16.87.123    266
        172.0.0.0        255.0.0.0      172.16.84.1    172.16.84.121     11

Now, what's the IP address of the firewall and what's the "other" gateway then?

The reply packets for the "external client" (10.6.4.125) should be coming in through gateway 172.16.87.5, as that's the way back for your them (default GW). If that's not the case, you have some form of asymmetric routing, which is a bad idea with statefull firewalls. But then you should at least be able to see the answer of your server on its other interface!

(17 Jun '15, 08:24) Kurt Knochner ♦

Do you mean to say the other interface 172.16.87.123 will send reply to 10.6.4.125? My filter was for both interfaces and via filtering 10.6.4.125 I could get only the response which I uploaded.

(17 Jun '15, 08:47) vjay

I can't tell you, because I don't know where the traffic came in. Can you please post the output of the following command on the server?

dumpcap -D -M

(17 Jun '15, 08:53) Kurt Knochner ♦
  1. \Device\NPF_{01FCA39F-F324-4E69-8846-98BC42FD040A} Intel(R) PRO/1000 MT Net work Connection Web Front 0 172.16.87.123 network
  2. \Device\NPF_{8BEF7088-B506-40A4-A1A2-D6D46A4BF9CE} Intel(R) PRO/1000 MT Net work Connection Web Backend 0 172.16.84.121 network

In network capture for both interfaces I only see traffic coming for 3389 from 10.6.4.125 IP but don't see any traffic going out from 3389 port to that IP Address.

(17 Jun '15, 09:04) vjay

as I suspected, the traffic in your capture file was captured on the interface with IP address 172.16.84.121, (see Statistics -> Summary. You'll find the ID of the interface there). Now, the default route will send out the answer through the other interface to gateway 172.16.87.5. This will result in a firewall seeing only parts of the traffic, or it will see the same session on different interfaces. In both cases a stateful firewall will deny to process the session.

Why you don't see the SYN-ACK in Wireshark, I can't tell you. Maybe there is something wrong with your capture setup. You should see the SYN-ACK and the ping response on the other interface (172.16.87.123).

There are two obvious solutions:

  • Add a route for 10.x.x.x/8 (or whatever subnet you need) to 172.16.84.1
  • Perform SNAT on the Firewall, and translate the address 10.6.4.125 to an address in the range of 172.x.x.x/8. Please ask your firewall admins, they should know what to do.

BTW: There is still a chance, that a filter on the Windows system blocks your frames, like the windows firewall or any other form of security software. Did you check that?

(17 Jun '15, 13:52) Kurt Knochner ♦

Yes, local firewall has been turned off and there is no antivirus software running on the server.

Unfortunately, Network Admin is stuck on the point that firewall is passing traffic but server is not responding, I will need some proof to get him make change on gateway or firewall. Is there a way I can capture response from other interface to 10.6.4.125? I do have whole capture without any filter available for both the interfaces but I don't see any response for 10.6.4.125.

(18 Jun '15, 00:12) vjay

To make sure you are really capturing on both ports, please start Wireshark with the following command:

wireshark -ni 1 -ni 2 -k -f "host 10.6.4.125"

Then start the ping and connect to the RDP server. Wait 10-20 seconds. Then look at the data. You should see the ping response now and the SYN-ACK.

If you don't see the answer packets, there is indeed a problem on your server and then you need to double check that nothing on the server blocks the frames (AV, IPS, Endpoint security, etc.).

(18 Jun '15, 01:11) Kurt Knochner ♦

Still getting same results. On Windows server also firewall has been turned off.

C:\Program Files\Wireshark>netsh advfirewall show currentprofile

Domain Profile Settings:
----------------------------------------------------------------------
State                                 OFF
Firewall Policy                       BlockInbound,AllowOutbound
LocalFirewallRules                    N/A (GPO-store only)
LocalConSecRules                      N/A (GPO-store only)
InboundUserNotification               Disable
RemoteManagement                      Disable
UnicastResponseToMulticast            Enable

Logging:
LogAllowedConnections                 Disable
LogDroppedConnections                 Disable
FileName                              %systemroot%\system32\LogFiles\Firewall\pf
irewall.log
MaxFileSize                           4096

Public Profile Settings:
----------------------------------------------------------------------
State                                 OFF
Firewall Policy                       BlockInbound,AllowOutbound
LocalFirewallRules                    N/A (GPO-store only)
LocalConSecRules                      N/A (GPO-store only)
InboundUserNotification               Disable
RemoteManagement                      Disable
UnicastResponseToMulticast            Enable

Logging:
LogAllowedConnections                 Disable
LogDroppedConnections                 Disable
FileName                              %systemroot%\system32\LogFiles\Firewall\pf
irewall.log
MaxFileSize                           4096

Ok.
C:\Program Files\Wireshark>

Don't know what could be the issue.

(18 Jun '15, 02:03) vjay

Still getting same results. On Windows server also firewall has been turned off.
Don't know what could be the issue.

I don't know it either, but if you still don't see the reply packets in the capture file (with the command I posted above), then I conclude that:

  • something on the Windows system blocks the incoming frames. I can't help you with that. Please ask your local Windows hero!
  • OR the server does not get an ARP reply for the Default Gateway. Do you see ARP requests in the capture file? What is the output of the following command?

arp -a | find "172.16.87.5"

(18 Jun '15, 02:14) Kurt Knochner ♦

C:\Program Files\Wireshark>arp -a | find "172.16.87.5" 172.16.87.5 00-23-e9-6e-09-87 dynamic

How can I check ARP requests in capture file?

(18 Jun '15, 02:28) vjay

Use a display filter of "arp".

(18 Jun '15, 02:45) grahamb ♦

You don't have to look for ARP requests, as the server knows the MAC address of the gateway!

So, my conclusion is: something on the server blocks the incoming requests from 10.6.4.125. You say, that there is no security software enabled, so I can't tell you what it might be. Please ask your local windows hero/guru to have a look at it.

(18 Jun '15, 02:46) Kurt Knochner ♦

Thanks Kurt for your extensive support. Really appreciated. I will check with Windows team.

(18 Jun '15, 03:04) vjay

You're welcome!

Hint: If a supplied answer resolves your question can you please "accept" it by clicking the checkmark icon next to it. This highlights good answers for the benefit of subsequent users with the same or similar questions. For extra points you can up vote the answer (thumb up).

(18 Jun '15, 03:08) Kurt Knochner ♦

@Kurt Knochner, I have added persistent route for the destination and it is working fine now. I thought to put an update so it may help someone in future.

(24 Jun '15, 04:42) vjay
showing 5 of 27 show 22 more comments
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×1,620
×22

question asked: 17 Jun '15, 04:39

question was seen: 8,973 times

last updated: 24 Jun '15, 04:42

p​o​w​e​r​e​d by O​S​Q​A