Surviving Electmageddon: Protecting against a wave of DNS outages
Update: Our own migration to multiple redundant DNS providers was completed at 10am Pacific time this morning, Friday November 4th.
Two weeks ago, DNS provider Dyn was attacked in a very large DDoS attack. IoT devices were used to send an overwhelming amount of traffic to Dyn’s resolvers which resulted in Dyn effectively being taken offline for hours. This took out Netflix, Paypal, Github, Twitter and many other name brand services.
The Dyn attack may have been retribution against a researcher from Dyn who collaborated with Brian Krebs – both of whom have been working to expose DDoS-for-hire and DDoS protection rackets. We think this explanation is more likely than it being a ‘state sponsored attack’.
The Dyn attack was the result of Internet of Things or IoT devices being infected with a botnet. At the time about 500,000 devices were infected and only 10% of them were used in the Dyn attack. The source code for the Mirai botnet that was used in the Dyn attack was released some time before the attack on Dyn and this allowed any attacker to build their own botnet and launch a large DDoS attack on any target.
Earlier this year, Russian hackers, codenamed ‘Fancy bear’ and ‘Cozy bear’ hacked into the Democratic National Committee which resulted in email leaks. This may have been an attempt to disrupt or influence the US election.
The US election is on November 8th, less than 1 week from now. Some candidates may benefit if fear, uncertainty and doubt are cast on the election itself or the results. Launching a massive DDoS style attack on DNS providers on November 8th would achieve that objective. It would take many services off-line, including news, exit poll results, official candidate websites, official announcement sources and services we rely on like banking.
We think that certain nation states may have reason to create this kind of disruption. We also think that malicious individuals with their own agenda may also try to create disruptions on November 8th. The Dyn attack demonstrated that by leveraging IoT devices and the Mirai source code, massive outages can be created by individuals or state actors.
In light of the above facts and the climate we find ourselves in, we would like to make a recommendation to owners of mission critical websites to help them weather the storm that may arrive on November 8th. We are suggesting a change in DNS configuration that is technically complex and also increases operating costs. We recommend this change for business or mission critical websites who have a technical staff they can call on to help them implement this.
We recommend that websites set up a secondary DNS provider via a different DNS vendor. By doing this, if your first DNS vendor is attacked, the second one will answer any requests for your domains IP address and your website or service will continue functioning as per normal.
To do this, you need to have a primary DNS provider that allows “zone transfers”. That means that the primary provider must give you the ability to authorize another DNS provider to copy or replicate all your DNS records from time to time.
The vendor you choose as a secondary DNS provider must have the ability to act as a secondary where another vendor is the primary. That means it must be able to do zone transfers from your primary DNS provider to replicate your DNS records.
Wordfence is moving to this configuration over the next few days and we’re implementing it with our current DNS provider, DNSMadeEasy as primary and with Verizon’s Edgecast DNS service as the secondary. We have verified that they can work with this configuration and they will give us the performance our customers expect.
The diagram below gives you a general idea of how a secondary DNS server keeps your website online if your primary DNS provider is attacked. Customers can’t lookup your website IP address and, rather than your website appearing offline, they are able to resolve your site IP with the secondary DNS server and connect to your website. DNS is more complex than the diagram indicates, but this gives you a general idea of how failover works from primary to secondary during a DDoS attack.
Finding a cost effective DNS provider that can act as secondary DNS to your primary provider can be a challenge. You may also have to switch primary DNS providers if your primary does not allow zone transfers to a secondary DNS provider. For example, Cloudflare does not appear to allow secondary DNS servers because they don’t allow zone transfers.
We did an audit of the top 10,000 websites (Source: Alexa), and out of 1832 domains that use Cloudflare, only 3 have secondary DNS configured on another vendor. We think these three are using something other than the standard zone transfer to secondary configuration because Cloudflare technically doesn’t support doing that.
Using a single DNS provider if you operate a mission critical website creates a single point of failure. As recent history has shown, this can leave you offline during a large DDoS attack. As the old Latin saying goes, if you wish for peace, prepare for war. We recommend mission critical websites make appropriate preparations in case we see a repeat of October 21st – and let’s all hope that November 8th comes and goes peacefully.
Comments
11:47 am
Hi Thanks for this article. It would be great if you could list a few possibilities here for primary and secondary DNS servers since we all will be doing the same research. We know that does not mean you guarantee their services, but it would shorten the research time for all of us immensely and we would be very thankful!
11:56 am
Hi Jennifer.
As I mentioned we're going with our existing provider DNSMadeEasy as our primary and we've confirmed they do allow zone transfers to a secondary. You'll find their prices are very competitive and they're very high performance with features like anycast and edge servers.
We're 99% sure we're going to implement this with Verizon Edgecast's DNS service. We're currently working with them. They'll be our secondary. They support being secondary-only. Their price is super competitive and they're one of the very few low cost, high performance providers that are OK running as secondary-only, and not being primary.
I'll post any additional research from our ops department that comes up.
I think most DNS providers - or in the case of Cloudflare, CDN/WAF/DNS provider - want to be primary and primary only to eliminate the competition. The thought is that they can build redundancy into their own systems. As we saw with Dyn, if a vendor is targeted, then all bets are off and sometimes it's best to just have another vendor as secondary. So when using or evaluating a DNS provider, I think it's OK to put some pressure on them to act as primary and allow zone transfers or act as secondary only. That's the way the Internet was designed to work. Not with vendor lock-in and single-points of failure.
Mark.
11:48 am
What is stop users replicating the DNS settings to the secondary DNS service manually? Is it really as complex as you say to set up an alternative DNS service?
11:51 am
As far as I'm aware you can do that and that's what we think the Cloudflare customers with secondaries did. But you then have to manually keep the two in sync which is a pain and something you can screw up. DNS is designed to have primaries and secondaries with automatic replication - so why not take advantage of that?
To be clear: It's not complex if you're technically minded. But if you're say, a realtor, or in a non Internet related job and have a WordPress site (like many of our users and audience), then doing this is time consuming because it requires a learning curve and some research.
12:01 pm
James Bamford -- one of the leading experts on cyber warfare -- does not believe it was Russia who hacked the DNC emails.
http://www.reuters.com/article/us-russia-cyberwar-commentary-idUSKBN12X075
12:18 pm
James provides a fine roundup of recent history in that article but there's very little in the way of providing his own evidence or research. Attribution is hard. Anonymity is easy. We all know that. So attributing an attack will always be a best-guess-level-of-certainty conversation. He also suggests policy towards the end of that article that I fundamentally disagree with. Cyberweapons are in fact very different to chemical, biological or nuclear because they fit onto a memory card the size of your pinkie nail with no supporting services or infrastructure. You can't detect them with satellite, inspectors and seismometers. So a disarm and enforce approach is absurd.
Thanks for the link Kevin, I hadn't read that. He clearly knows his cyber history.
12:04 pm
What if we use CloudFlare? Do we need a 2nd DNS Zone?
12:18 pm
You can't get one. They don't support zone transfer to a secondary DNS provider.
12:05 pm
Secondary DNS is a smart move, but pick your secondary carefully. I used one with great success for years, but then "a hard drive crashed" and their whole operation was down for a week or more... an unforgivable sin in a world where redundant DNS is just not that hard. Only then did I understand how mom and pop their organization was.
Secondary DNS services are one of the cheapest services on the Internet... money well spent.
12:19 pm
I appreciate your security tip, but I believe the statement "[s]ome candidates may benefit if fear, uncertainty and doubt are cast on the election itself or the results" is a reckless and unnecessary comment.
I don't think anyone "benefits" from an attack on the Internet. We need to trust your security comments on the Internet are based on technical ways to protect our web sites, never any political agenda. I would urge you to reconsider that comment about "some candidates" benefiting. It is counterproductive and unnecessary.
12:35 pm
For details on my political agenda, which way I'm voting and why it's best to not read too much into my technical blog posts:
https://www.youtube.com/watch?v=xfr64zoBTAQ
3:14 pm
He tells it like it is Jeff.
12:20 pm
Kevin Strom, if you read the article you cite, it's an overstatement to say he "does not believe it was Russia." His article only calls for caution, saying the evidence is ambiguous; he doesn't affirmatively say it wasn't Russia. The preponderance of independent experts seem to think it's at least likely it was Russia. Also, it's quite possible the U.S. government has evidence to which we are not privy.
12:26 pm
Agreed - he's basically playing the old-wise-wardog card and saying "Lets not be so hasty" without actually doing any work. It's an opinion piece.
Also agreed most evidence points towards Russia.
I'd add though re US secrecy: "It's classified" is a really useful designation if you are lazy. I believe that secrets are necessary. However I think they remove accountability, especially in government where departments are accountable to the public that funds them. And so when folks (And no offense meant here Steve) assume a level of competence or ability might exist because something is secret, it makes me grind my teeth.
Mark.
12:26 pm
I'd like to suggest a lesser strategy for those who cannot afford or for other reasons cannot accomplish using multiple DNS Providers.
At the very least, don't choose a DNS provider that keeps all their eggs in one basket.
Dyn, Godaddy, EasyDNS, DNSMadeEasy, Network Solutions are relatively easy to take down because they have a relatively small number of DNS servers. Overload them and you take down large swaths of the internet in a single blow.
Cloudflare is a little better, each customer gets a separate pool of DNS servers, and these are multi-homed. That said, while Cloudflare has weathered every DDOS attack that has hit them so far, it is still a relatively small pool of DNS Servers, so it is theoretically possible to take them down (or at least take down regional DNS providers).
One DNS provider that should be able to weather the DNS Storm pretty well is AWS Route 53. Every single domain on route53 has a unique set of 4 nameservers, and these are on disparate domains (not just disparate subdomains). Those nameservers in turn are geographically distributed and anycasted. Because of this it becomes very difficult to completely take down Route53. You might be able to take down a single website by overloading the 4 dns servers for that domain in a single region, but taking down all of Amazon's customers at once would be extremely hard.
Of course, as you point out, the best solution is to use 2 completely different DNS providers if possible. Some DNS providers make this easy in one way or another. EasyDNS for example has a service they call EasyRoute53 where every DNS change at EasyDNS makes the api calls necessary at AWS to update the Route53 zone at the same time.
The reason I say this is because there are legitimate reasons why you would not be able to use 2 dns providers. If you use a Load Balancer or CDN that requires dynamic resolution of the bare domain (like Cloudflare or AWS's Cloudfront or Elastic Load Balancer), you may be tied to a dns provider for that reason. The benefits of using such a system may outweigh the benefits of using multiple dns providers under most circumstances, and in those cases, you can at least choose the "lesser evil" of using a dns provider that at least make things difficult for attackers.
12:30 pm
Seriously guys? I think you are just targeting CloudFlare and trying to make them appear less useful simply to get pro signups. This is the second posting I have seen like that.
CloudFlare uses secondary and tertiary DNS - they just happen to own the locations. Their network is worldwide. Protecting DNS is NOT about "different vendors," but about different locations. There are some "secondary" vendors out there that happen to be IN THE SAME DATA CENTER. They would NOT prove helpful to use.
The premise of using secondary DNS itself is valid, but if your current vendor is world-wide and correctly setup, then there is NO issue. We have 127 sites on Cloudflare in 5 different physical locations (US, Europe, New Zealand, UK, etc.) and NOT ONE was affected by the Dyn attack.
While I find most of your Blogs useful, this one was a bit over the top ref. CloudFlare.
Joe C.
Chief Engineer, Dead Parrot Software Inc.
12:36 pm
It came up in our research. Couldn't ignore it. We're not targeting them specifically.
12:46 pm
I'd also add a few comments:
It's vendors who are targeted and they are targeted in multiple locations when it happens. So we're suggestion that you mitigate this by diversifying the vendors you use for mission critical infrastructure where possible and where costs allow. Since DNS is so cheap and running a secondary on another vendor is low risk compared to, say running hot failover, we're suggesting owners of mission critical sites do this.
I'd also add that I don't like vendors who opt to not provide this service and therefore benefit from a vendor lock-in. It helps them, not the customer.
Dyn was targeted, not Cloudflare.
And even though Cloudflare was not the focus of the attack, they were affected.
DDoS is a hard problem and there are no solutions to this issue yet, no matter what solution providers claim. Think you have DDoS through a cloud WAF provider, they'll just target the origin. Got protection at the origin, they'll target infrastructure. Protecting everything, they'll just ramp up the volume of the attack until your DDoS mitigation simply can't keep up.
Ask any DDoS mitigation provider world-wide if they've solved DDoS and they'll answer: "Sortof".
As I mentioned in the post, only 10% of IoT devices compromised by Mirai were used in the Dyn attack. It was greater than 1Tb in volume. The math is fairly easy, but if you love your vendor so much that you don't want to diversify, at the cost of being taken out if that vendor is targeted, I'm not sure I understand but I wish you the very best of luck.
Mark.
3:12 pm
Thank you Joe for chiming in with this. We are just getting started and have 5 Cloudflare sites - soon to be 6 - and, as with you no problems.
For a minute reading these articles I was getting a little concerned. Looking at the two "attack" maps on the Cloudflare article though I saw that I was in one of the hardest hit areas of the second wave. But again no problems.
We use Wordfence Premium as well and am still happy about that. But neither have I found reason to be concerned about Cloudflare.
Advice? I learned long ago when I read the article by Yoast condemning any thought of the use of a slider as shameful and dirty (I paraphrase here, but that is how the article left you feeling), advice always has to be taken with a grain of salt and followed up by further reading.
3:24 pm
Hi Joe!
Of course cludflare was not affected by the dyn outage. Had cloudflare been affected it would have been the cloudflare outage. They were not the target and hence were not affected.
Although cloudflare is a good solution and works for a lot of sites, it is not a catch all solution and has major issues. I'm glad cloudflare works for you but don't praise them to much, they are not the solution for everyone and especially not for every problem.
Regards
John
Senior IT Analyst
Company
1:42 pm
What is preventing your second DNS from going down after a attack? Why not just stick with Cloudflare who are designed to keep your site online even during a attack? That's what you pay them for! Even if Cloudflare's DNS servers did get over loaded they automatically move you over to another cluster. I understand what you are saying but both DNS providers needs to have a good DDoS protection. I can't think of any other DNS providers that has more clusters of servers world wide.
1:58 pm
Dyn, UltraDNS, Route53, dnsmadeeasy, verisign all provide edge servers for DNS for super low latency and have servers around the world with a huge number of points of presence. Cloudflare are a newcomer to the space.
I'll say it one more time: Vendor diversity. Hey that's actually catchy. "Vendor diversity" - that's a great phrase to describe what I'm advocating for here. If you can employ vendor diversity at low cost and with low risk, why on earth wouldn't you? Especially given the current state of DDoS protection and how quickly attack size and type is changing.
1:43 pm
I've been a hosting provider for 20 years. We've always run 2-3 separate name servers, in 2-3 separate locations under separate providers.
There really is not a "priority" when it comes to name servers. On your domain name registration, you can enter 2 or more name servers. When a DNS query is made to the root name servers, ONE of the nameservers on your domain is used AT RANDOM to lookup the IP address.
In other words, the name server listed first on your domain name registration DOES NOT get used more than the second or third name server listed on your domain.
So, the WHOLE point of this discussion is a suggestion that the nameservers on your domain name registration SHOULD BE FROM MORE THAN ONE COMPANY, as if not, you have a single point of failure.
What I think most people are confused about is that you CAN NOT setup one name server on your domain to be used more often than another.
Then why are they called primary and secondary DNS providers? This is 100% about keeping the DNS records in sync. DNS has settings that allow one DNS server to be a "slave" and sync the information from the "master" server, period. In other words, the "primary" and "secondary" labels are ONLY for how the data is kept in sync, it has NOTHING to do with which server will be used to do lookups. (Several years back with was a big scare about "DNS poisoning", which used a flaw in the DNS syncing setup to change the DNS records, which could allow domains to be pointed to a different host.)
Bottom line, it's not priority, it's redundancy. So John_B's comment about keeping them in sync manually is correct, so if you are willing to keep them in sync manually, you can. (My guess is there are many people reading this who VERY rarely change their DNS settings, which means there is rarely, if ever, anything to keep in sync. I'd also guess that many readers have their domain registered with one company, and hosted with another, so they may already potentially be able to have one of the DNS servers on their registrar, and the other with the host, without adding ANY cost.)
I hope that is helpful.
Peter Janett
New Media One Web Services
http://www.NewMediaOne.net
http://GeoDigs.com
support "at" newmediaone.net
(303)828-9882
Add us to your address book:
http://www.newmediaone.net/New_Media_One.vcf
1:52 pm
Thanks Peter. Very interesting point regarding being able to do this at no cost using the host and registrar.
Mark.
1:55 pm
The DNS protocol is prepared to switch to the secondary server when the primary does not respond. I think the problem here has been that the primary does not stop responding (but rather a delay occurs in the response due to DDOS) and therefore do not change to the secondary. To solve this problem I think is not as simple as implementing a secondary DNS server, but rather load balancing between the two DNS servers, so as to detect when the primary is overloaded and switch to the secondary, but that is beyond the DNS protocol. Perhaps that is why they have chosen to attack the DNS protocol and not others probably already incorporate load balancing.
2:00 pm
This is incorrect. Peter Janett's comment gives a good summary of how DNS actually works.
3:33 pm
Hi again, I'm not focused on the fact of primary/secondary nameserver chosen (it is random, ok), but the fact that there is no possible loadbalancer (out of the box) for the nameserver election when your first query a domain. Therefore the dns protocol is more likely to cause problems than others (that can be put behind a load balancer; in the case of Netflix for example surely they have loadbalancers for web/media servers in its internal networks). They have therefore focused attacks on DNS servers.
Configuring a secondary DNS server (or tertiary, ...) you mitigate the problem but not solve it.
5:18 pm
Additional good reading on this topic (I'm not affiliated with this site):
http://blog.easydns.org/2010/08/19/dos-attacks-and-dns-how-to-stay-up-if-your-dns-provider-goes-down/
9:14 pm
Interesting. Let's wait and see how it pans out.
5:29 am
This may be a tiny bit out of scope of the discussion (DNS), but while we are talking about attacks let's not forget to mention backing up your website. Have a complete full backup of your website copied to somewhere not accessible by the internet. An example for a small website would be a hard drive that you keep in a safety deposit box. Make it a habit to update the offline copy on a regular interval--how often depends on how much you're willing to lose if there's a disaster, and how much it will cost your business bucks-wise.
11:08 am
Smart suggestion, Susan. With all the "big talk about big sites" I overlooked the most basic precaution for those of us with small business websites. Thank you!
6:50 am
IF attacckers attack first dns and second one ? :D