Hello, I have a basic DNS setup (defaults) and one forwarder setup to my ISP. (the box has 2 NICs, only one is enabled) My problem is that DNS requests are timing out (2 seconds) intermittently on various the machines looking to this server. There doesn't seem to be any patterns or triggers... DNS requests succeed when made by one machine while failing from another. Then they might switch... A very odd thing happened when I started to do packet capturing on the server using Wireshark.... The moment I started the capture, the problem went away. I confirmed it, first, by letting the capture run all day... no DNS requests failed. Then I stopped the capture and loaded YouTube on one of my machines... the DNS lookup failed. The moment I started the capture... the page refreshed and loaded. How can this be and how do I end my server's dependency on this tool ;) Thank you! -Scott asked 03 Dec '15, 19:34  CogitoErgoBibo |
2 Answers:
Please capture with promiscuous mode off. I would suspect an issue with arp so some DNS queries may be sent to the proper IP but wrong MAC, and promiscuous mode allows such packets to be seen and processed. answered 03 Dec '15, 22:28  sindy showing 5 of 18 show 13 more comments |
Well.. so far so good. I'll mark yours the answer after a day goes by with no problems just to be sure. You setup DRAC as part of the initial machine setup, but the license to use it isn't included in the price (actually I couldn't find a place to even buy one)... it asks you for the machine's IP address but I didn't see a place to specify which NIC to use. So, yes, it assigned itself to the same NIC I happened to use. Once I disabled the "DRAC NIC", the problem seems to have been solved. I'll update the thread as I mentioned... thank you for all your help! answered 07 Dec '15, 12:30 CogitoErgoBibo |
well the other strategy may be to look at the promiscuous capture and determine which MAC appears that shouldn't ;-)
I sure will try both suggestions, thank you.
I am wondering, though, why would all DNS queries succeed when the capture is on, yet fail when the capture is off?
The suspicion is that although the DNS requests (or responses from the master DNS server, whatever) are sent to proper IP address, there is some issue in MAC address resolution and thus those which get lost while you do not capture are not processed by the DNS daemon because a network card normally ignores packets for other MAC addresses than its own and does not pass them to higher processing layers.
When you capture in promiscuous mode, you remove this barrier by telling the network card to ignore destination MAC and give to the upper layers every packet it gets.
Got it! Thanks.. I'll run some tests now.
It seems you were right! When I turned off promiscuous mode, the next DNS request from the client failed. So, what now? I've been in tech for quite a while, but I've never encountered an ARP issue!
Can you give us a more detailed layout of your network? Hosts, networks, interfaces, addresses, that sort of thing.
Now follow Jasper's suggestion: if you have saved the captures done with promiscuous mode on, use display filter like
ip.dst == 172.22.238.241 and eth.dst != 00:e6:42:77:7e:71on them, where the IP address and MAC would belong to the used NIC of your DNS forwarder. It should show you some packets which came with proper IP and wrong MAC.My audacious guess would be that your DNS forwarder occasionally responds to ARP requests coming through the used NIC with MAC of the other NIC.
Sure... Pretty simple:
Cable Modem->Firewall Appliance->Gigabit Switch
Off the switch: - One Windows SBS 2011 (server 2008 R2) (new install, *DNS server) - Several desktops/laptops/tablets with various OS's. - Linksys EA8500 wireless router configured as an AP.
I applied the filter as you suggested sindy... I found several hundred packets. Wow! I've never heard of this... what now? The dual NIC's are onboard... Is there a standard solution to this? (other than my hope that I can disable the non-active NIC in BIOS)
I don't know any "standard" solution, and I still don't know what exactly happens.
So first of all:
can you confirm that the other MAC you can see with my display filter is really the MAC of the other NIC? It is not clear from what you wrote so far.
if so, is it "not used" (i.e. the cable is not connected) or "disabled" in Windows? Go to the "Center for networking" window and look whether the unused NIC is in full colour or grayscale; if in full colour, right-click it and choose "disable", then try again (but bear in mind that the records in ARP tables have expiration time of minutes so don't try too quickly)
If the above fails, the next step would be to try the BIOS, and if BIOS would not offer any means to disable the card, we'd have to invert the strategy. It should be enough to enable the 2nd NIC, give it an IP address from the same subnet like the other one, and connect it to the switch. This way, the IP stack would get the packet through one NIC or the other, and as it doesn't (and actually cannot) check whether IP and MAC addresses match each other, it would process it in any case.
Just got back from lunch so I'm going to check to see what the other address is...
As to the NIC, when I first setup the server I disabled the second NIC, but when I looked this morning it was enabled. I disabled it, but I have wait until Windows Update completes before I can reboot. (assuming this is necessary since just disabling didn't do the trick)
(thank you so much, again, for all your help)
I was trying some filters to make the search easier and I noticed something... In the capture I made with promiscuous mode turned off (failed name requests), there was no traffic with a destination port of 53. All the name resolution requests coming from the client were made over NBNS!
Well, as I think about it now, maybe I've overestimated your knowledge about how MAC addresses should look like, and I forgot to include broadcast MAC address into my display filter. So the new version would contain also
and eth.dst != ff:ff:ff:ff:ff:ff, and the ff:ff:... must not be replaced. If you add this to the end of my original filter, do you still get hundreds of packets which match the filter condition? And what is the "legal" MAC and the one which is seen as destination in the packets shown with the extended filter?Thanks sindy... as I understand a DNS inquiry, the first thing that happens is that the client makes an ARP request on the IP address of the DNS server. Once the ARP request comes back with the MAC address, the client uses that address for the DNS query. Is that correct?
I using MS Network Monitor before a friend told me about Wireshark. From the client I captured a successful resolve request, but I think I see the problem.
When the client sent the ARP request on the IP address of the server, there were two responses. With two active NICs, I figured this wasn't so odd. However, the first address returned was wrong... but oddly wrong (at least to me).
The two NICs in the machine have almost identical MAC addresses... The first five octets are the same (understandable), with only the last octet differing. One of the ARP responses was correct... it was that of the active NIC. However, the second not that of the disabled NIC.... the first five octets were the same, but the last octet was off by two. (Lets say the disable's NIC's last octet is 42... this one ended in 44.
In a MS Net Monitor capture of a failed attempt, I can see that the DNS query was made using the ::::44 address. The one that doesn't exist.
Mostly yes, except that
it applies for any packet you want to send to a given IP, i.e. is not limited to DNS inquiry,
it only happens if the MAC address for a given IP is not available in the ARP cache. So if you are sending packets to a given IP address often enough, the ARP request is only sent once in the beginning. Depending on the operating system, if packets are coming from that IP often enough, you usually do not need to send an ARP request either because the incoming packets' IP and MAC are stored in the ARP cache.
The important part is that you get two MACs for a single ARP query, and either the first one to arrive or more likely the second one to arrive is ARP-cached and used as destination for outgoing packets to that IP.
I would say it is time to use a bigger hammer. I'd disconnect the DNS server from the network completely (including the IPMI if present and including the WiFi adaptor if it has one), and connect a notebook with wireshark to its NIC which is normally in use, using a direct cable (no switches). At the notebook, this must be the only active network interface too. The IP of the DNS must not be set as default gateway at the notebook. When the ethernet link comes up, start a Wireshark capture on the NIC of the notebook, and in command line send "ping IP_of_the_DNS". You should see an ARP request and one or two answers.
If you still can see two, it is a Windows Server OS issue (with one exception); if you can see only the correct one, I assume one of the following:
the IPMI was connected to the switch and has got the same IP like the NIC.
your wireless AP is in bridge mode and the wireless card of the DNS server was active before but Windows firewall did not let the requests in through it,
(least likely) there is another machine on your network which has been manufactured right next to the one running the DNS server (so it has got the :::44 MAC) and has the same IP.
The exception mentioned above is the following: To make life funnier, some servers can be configured so that the NIC and the IPMI use the same physical connector. So before opening a ticket with Microsoft, please change the IP of the server's NIC and do the test above again.
Are you familiar with Dell's DRAC? It's Dell's platform based on IPMI. I think you are right that this might be the root of the problem because both address come from the server. I have to pull the ol' monitor out of the closet tomorrow so I can get into BIOS and disable it.
If it ends up being the case... at least I learned a bit. And also, I think I may have a snoop on one of my Windows 8.1 machines... it sends out near continuous ARP requests!
I'm not (my IPMI experience comes from IBM/intel servers), but I'd say it is either a really funny coincidence (i.e. that you've used the same IP address for the NIC like was set on the DRAC and the DRAC is connected to the LAN, either by its own cable or because it uses the same Ethernet connector like the NIC), or a bug of the server's firmware, and in the latter case, it may not be curable by disabling the DRAC in BIOS settings but you may have to remove it physically.
But I'd think Dell provides an utility application allowing you to configure DRAC while the main operating system is on, so maybe you should try to use it to check the DRAC's IP (and possibly also check its MAC once there).
Or simply start by assigning to the NIC a different IP than it has now, as I've already suggested, to avoid the supposed address conflict with the DRAC.