Lessons Learned from the DynDNS DDoS

As everyone probably knows, DynDNS was recently hit by a massive DDoS which in turn caused large sites to be either nonresponsive or extremely sluggish. Dyn DNS was hosting records for these organizations when an application layer SYN flood attack against their DNS service brought them to their knees. The attack caused legitimate DNS requests for these sites to be “lost in the mix” with a steady flow of garbage requests saturating Dyn's DNS service. After watching the attack play out, I had a few thoughts on the subject I’d thought I’d share. 

I’ve personally fought DDoS attacks in the past and they’re not fun. To be bluntly honest, they’re a pain in the butt.  Many times they come out of nowhere and it’s an all hands on deck situation when the flood starts. But after seeing the recent attacks on Krebs, OVH and now Dyn, it seems that everyone on Twitter has recently become a DDoS expert. It takes some skill and most importantly experience when dealing with DDoS attacks, so let’s not take this subject lightly. We need to learn from our mistakes and the incidents of others to achieve the best security we can possibly offer. Let’s not just start being a Twitter warrior with nothing to back it up. Okay, I feel better now. 

This being said, now that we all know DDoS is a huge issue (because the media doesn’t lie, of course!) those who work in the security field can’t plead ignorance anymore. Just because your industry doesn’t normally see DDoS attacks doesn’t mean they won’t pop up and smack you in the face now. With the tools and vulnerable systems to create massive botnets we might only be seeing the beginning of what’s in store. Everyone in charge of security needs to start the process of creating a DDoS runbook today. This needs to become a table top within your incident response plan. Incident handlers and groups outside of security need to understand how to handle DDoS attacks when they occur. The last thing you want is an attack to occur without any preparation. The Dyn DNS team did a great job explaining to the public how the attack was being handled and gave frequent updates through this site: www.dynstatus.com. This is important during an attack that knocks you off the grid. Communication is key during this time, especially to your customers. 

Another thing to consider is how a DDoS attack will be mitigated. With attacks cranking in at over 1Tbps there is no on-premise DDoS mitigation appliance in the world that’s going to handle the load right off the bat. Not only will they not physically handle the load, but the ISP’s will have issues fulfilling traffic of this magnitude. The current infrastructure just isn’t designed to handle this amount of traffic traversing its network. The best method of mitigating these services isn’t with onsite DDoS appliances, but with cloud providers like Akamai (formerly Prolexic), Cloudflare, or Google Jigsaw. They’ve positioned their network to be resilient, with multiple scrubbing centers throughout the world to absorb and filter the malicious traffic as close to the source as possible. By using anycast and having traffic from customers directed to them via BGP, these cloud providers make sure they don’t become a bottleneck and allow customers to receive large amounts of bandwidth via proxy. I personally feel this is the only way to efficiently defend against the volumetric  attacks we’ve seen this past month.  Also, Colin Doherty was announced as the new CEO of Dyn this October 6^th. He was the former CEO of Arbor Networks (a company selling and specializing on premise DDOS solutions). I don’t know if this had anything to do with the situation, but it’s interesting. If anything, hopefully his experience in the industry helped with the mitigation.

For the cloud providers who are absorbing and mitigating DDoS traffic on their networks, they’re going to have to expand their available bandwidth quickly. Many cloud based DDoS mitigation providers need to have bandwidth increased by a certain percentage each time they see an attack increase. They all want to be a particular percentage higher than the largest DDoS attack on record. This is because they too have to scale towards the attacks as they come in. They’re not only dealing with the one large attack occurring today, but possibly three more like it tomorrow at the same time. These providers need to keep a close eye on bandwidth utilization and attack size monthly to keep up with the growing botnet sizes. 

I’m not sure what happened with the Dyn DNS attack from a mitigation standpoint, but it’s a good opening for customers to start speaking with their third party vendors on incident response; especially on DDoS. Many third parties say they have DDoS prevention, but how? Is it home grown? On-premise? In the cloud? These questions need to be answered.  Also, if a DDoS hits a SAAS provider will all clients go down? These and similar questions need to be asked of your cloud providers to validate your hosted services will be available when needed. 

IoT will continually be an issue going forward when it comes to DDoS. I don’t see anything in the near future putting a stop to the abuse of IoT systems on the internet.  In Brian Krebs latest article he mentions Underwriters Laboratories and how they’ve been used in the past to become a sign of approval for devices going to market in the electronics field. I think there does have to be something similar in the future that assist with reviewing the code of appliances before being put onto the internet. At this point I’d settle for standard OWASP top 10 type scans, but would to see static analysis testing done for vulns. I don’t know how this will work with systems overseas, since most of the Miria botnet infected DVR and IP cams from a Chinese company named XiongMai Technologies. Either way, we need to at least follow standard security practices of password management, patching and secure coding when it comes to IoT devices. This isn’t rocket science, especially when many of these systems were using default hardcoded passwords and being logged remotely with telnet. Sigh.

My concern with botnets of this size is that someone’s going to create multiple IoT botnets quietly and unleash something with traffic limits that can’t be stopped. There are other vulnerable IoT systems on the web which will eventually be found, but what if this time they weren’t used right away. What if the creator keeps finding other vulns in different systems and ends up with a botnet-of-botnets with enough power to overwhelm even the largest DDoS cloud providers. Now take this a step further: What if this was then used for political or terrorism?  I know this sounds like fear mongering, but it’s a valid concern. In this case, people would die or be hurt in the process. This is a concern of mine with the amount of insecure IoT devices being connected to the internet today. It might seem farfetched, but it’s no longer outside the scope of reality. The Miria botnet was seen as being used in the Dyn DNS attack (by Flashpoint, L3 and Akamai), but it seems that there were other systems being controlled in the botnet too. It just seems that there a never-ending pool of IoT devices that attacker can select form at this point. 

As of right now I haven’t seen any official motive for the attack, but there doesn’t always have to be one. I saw people mention that it’s a test for the Unites States election, WikiLeaks took credit for it due to America pulling Assange’s internet, internet activists blaming Russia, etc. Either way, everyone in security needs to be prepared for these attacks and if you’re not already planning now, at least start thinking about it. We’re no longer given the luxury of being comfortably numb.