Hi Folks! This is my second question to the forum, and it's a little different in the sense that I don't know that posting the capture file will help - at least here. I had been troubleshooting a problem at work reported as slow DNS response, started looking through packet details, and was digging into failed DNS recursive queries when I saw other failed queries that were very strange. They used the domain suffix for my organization, but the first part of the domain name appeared to be gibberish. Something like, qe4l7rius9adfe.mydomain.com. They came three at a time, a few nanosecs apart, at variable times throughout the day. From my first observation subnet, they were recursed from other DNS servers. I moved my observation point to the source subnet and checked again, identified a subnet used for VDI systems to support offshore resources, and then saw additional DNS servers and other hosts requesting similar bogus names. We temporarily blocked the VDI subnet (the majority of the activity), and then saw additional sources for the requests. I had read something in one of my Wireshark Kindle books in a DNS section about how this could be possible malware activity. I thought the referrence was to DNS Fast Fluxing, but after googling around I found security articles for DGA that actually were a better matching description for the observed behavior. I had created a DNS hostname columun that helped make them jump out in the packet list, as these strange failed DNS names weren't evident in any kind of summary view, just in the packet details. Tedious to review by drilling into each packet, and hard to isolate from the rest of normal DNS activity. It's really a miracle that I stumbled over them at all, as upon reflection I don't know how you'd otherwise screen for or detect this activity. I found a great wiki link describing the DGA behavior, and some other articles that got very deep really quick. Following one seems to indicate that the best form of detection requires determining the encoding algorithm, and then somehow filtering or looking for that. Sounds like a job for a security contractor to me! I notified my managers and forwarded the tracefile to our ONE security guy, and haven't heard a word from them since. I guess what I'd like to know from anyone in this forum is, have you seen DGA before (I couldn't find another question here about it), and if so - how bad is it, what's the most effective method to block it? If you read the wiki, this method is part of a botnet communication strategy. What I couldn't determine is, does this confirm that I have bots on my network that are using these DGA names? Is there a simpler/easier way to remedy other than using security consultants to reverse engineer the algorithm (used to "type" and source the attack vector), can I do this usuing Wireshark, or should I advise my company to get the Feds involved? We're processing HIPA data, and I'm greatly concerned - I'm just lacking context and experience with this type of issue. Thanks! asked 21 Jun '15, 18:08  Lucidcryotank |
One Answer:
This could be Google Chrome checking if the owner of the DNS server is modifying DNS requests, for example, by using wildcard domains to match requests for non-existent domain names. See the InfoSec article Google Chrome and (weird) DNS requests and also the Wikipedia article for Wildcard DNS record. answered 21 Jun '15, 21:44  Jim Aragon Interesting. I'll have to dig deeper into this info. How could you verify if it was Chrome activity? Is there a log on the host running Chrome that I can correlate the event to the DNS query sent from the host IP in the query? Chrome is in rare use, but present. I'll look more closely at how to disable that and check one of the querying hosts after reconfig. Thanks for the info! If this is the case, our one security guy can stand down from his efforts, saving all of us time. (21 Jun '15, 22:44) Lucidcryotank Not sure why the example trace shows the same DNS name twice in that sample capture. I only see one request & response for each of the 3... (21 Jun '15, 22:52) Lucidcryotank Actually I was wrong about the latter. I'm going to anonymize my trace and post it. The names vary in length, and do not contain any numbers or special characters. (22 Jun '15, 08:01) Lucidcryotank Here's the anonymized trace, ignore the malformed packet info - it was generated by Tracewrangler because the original packets were sliced at capture to 92 bytes and were originally notated as "packet size limited during capture". (22 Jun '15, 08:22) Lucidcryotank I do see now where the same bogus DNS name is tried on multiple domains. (22 Jun '15, 08:25) Lucidcryotank Some of this activity is again recursion between DNS servers. All of the individual source hosts are again on our VDI network. I'm going to find and talk to the VDI admins to see if Chrome is part of their desktop build. (22 Jun '15, 08:44) Lucidcryotank Chrome will try the same names using each locally configured DNS suffix. I'm not sure how you would confirm if this is Chrome. I'd start by capturing on a PC that has Chrome installed. Make sure Chrome is NOT running. It normally runs in the background even after all Chrome windows have been closed. On Windows 7, after closing all Chrome windows, click on the "Show hidden icons" icon in the System Tray, right-click on the Chrome icon, and select "Exit." Start capturing with Wireshark, then start Chrome, and see if the DNS lookups generated by Chrome seem to match what you saw in your earlier traces. I see Chrome generate three different domain names, and try each one multiple times. (22 Jun '15, 08:52) Jim Aragon Is the number of suspicious machines increasing? I can see 3 different IP Addresess? And if you really think that some machines are infected, why don´t you isolate them asap? But it looks to me more like the things in Jim Arragons link then this here: DGA Case Study (22 Jun '15, 12:56) Christian_R That's a great diagnostic plan, except the machine that's running is a VMWare VDI (virtual desktop interface?), and I don't think I'll get them to put wireshark in it or allow me to install it - but I'm not sure it's necessary. I should be able to observe the same event regardless of whether I'm on the workstation or back on the subnet where the DNS server is. As long as I know the VDI's IP, I should be fine watching the behavior on the DNS server subnet when I disable and re-enable Chrome. I think this is a solid method, maybe I'll even get a VDI of my own and see what permissions I can have. The number of infected machines doesn't necessarily keep increasing, but it certainly changes as VDI's come and go throughout the day. I bet if I were to look at enough data to see a trend, any increase in this activity would actually be the offshore resources using them during our CST off-hours. There may be a few local VDI users (devs) during our business day, Chrome isn't in the corporate workstation standard build, but several users with local admin have installed it. I also think Jim's explanation is more of a match and plausible explanation than DGA now. However, knowing how screwy this is and how it looks, I'd kinda like to see examples of DGA traffic and compare them. I've notified my security guy, and he's relieved. We did temporarily shut down a couple of the ESX hosts serving the VDI's, but the devs had to have them back. And like I said, our security department is ONE guy. If this isn't DGA (looking unlikely now), I'm glad that what little screaming wolf I did didn't seriously disrupt our business. As I couldn't exactly identify what the impact would be if it truly were malicious, I didn't make a federal case out of it - I was still trying to determine if it should be a big deal or not. As there was already a guy assigned to look at it, I kind felt that I had played my part as a network engineer, and had notified my manager and top execs. My biggest mistake was probably not posting it here sooner! Like I said though, I couldn't find it in any other question, and I was already working on my other post. I guess I thought my company would at least check with the security consulting group for their analysis (kinda wanted to see how that worked if it had been real), but again - now I'm glad they didn't. (22 Jun '15, 21:15) Lucidcryotank I was having trouble with the concept of infected VDI instances as well, and the entire offshore team uses them. (22 Jun '15, 21:19) Lucidcryotank I'm also in the middle of a Riverbed eval I built for NetExpress, VShark and Packet Analyzer. I have continuous VLAN traffic captures (packet sliced - on both the VDI and DNS server subnets) running on multiple ESX hosts in a cluster, with several days of history. I can look at the data there at any time, which to me is better than trying to run Wireshark on a VDI anyway. I'm still learning the best methods for using all of the tools in the suite for analysis. I do believe that studying Wireshark has improved my analysis efforts overall, more than just overlaying all of the interesting Packet Analyzer filters. The trick to that seems to be finding the incidents of interest quickly using PA, then zooming into the packets with Wireshark. If we had more engineers and I wasn't trying to perform a network redesign and new hardware implementation at the same time, I'd have more availability for pure analysis. (22 Jun '15, 21:31) Lucidcryotank I've been provisioned with a VDI client. I'm going to try and recreate the event and make a new observation. (24 Jun '15, 16:34) Lucidcryotank showing 5 of 12 show 7 more comments |
Also, EVERY one of these DNS names is unique, and I only saw each one once.