Endpoint vs Cloud Security: The Cloud WAF Bypass Problem
Earlier this year at Black Hat 2016 there was a lot of buzz around “endpoint security”. In this post I’m going to explain a few issues with a cloud approach to web firewalls. Then I’ll explain the benefits of endpoint security and why Wordfence takes an endpoint approach to protecting your investment.
The Cloud WAF Bypass Problem
Cloud firewall providers like Sucuri and Cloudflare have servers that live out on the internet. When you configure a cloud firewall (or cloud WAF), the provider will ask you to point your website at their servers by making a DNS change. Once you’ve done that, the cloud provider configures their servers so that when your web traffic arrives there, it is filtered using their firewall rules.
Once filtered, the traffic is forwarded over the public Internet to your website. In this configuration your original website is known as the “origin” or your “origin server”. That is where your site originates from and is actually hosted.
Your origin server may be hosted at Bluehost, Hostgator, Godaddy, on your own server or with one of the many other hosting providers.
In theory once you have configured your website with a cloud firewall (or cloud WAF) you are protected by it. The expectation is that attackers will try to access your website using your domain name of www.example.com, they will be pointed to the cloud WAF and their attacks will be filtered out.
In reality, if an attacker can discover your origin IP address, they can simply bypass the cloud WAF as the diagram below shows:
In the case of a cloud WAF, you aren’t actually “behind” a firewall because the server is still on the public internet. Anyone on the net can access your server directly if they discover your origin server IP address. Knowing your origin server IP allows them to simply go around the cloud firewall provider and attack your origin server.
Cloudflare have acknowledged this problem in a blog post and they provide various suggestions on how to keep your origin server IP address “secret”. They also explain how to access your origin server directly for testing if you have the IP address.
We refer to this problem as the “Cloud WAF Bypass Problem”.
The Cloud WAF Bypass Problem is Well Documented
This issue is well documented by many security bloggers. Tools like CloudPiercer.org and CrimeFlare exist to help attackers bypass a Cloud WAF.
CrimeFlare lets you look up a Cloudflare customer’s origin IP address or download an entire database of 1.5 million Cloudflare customers and what CrimeFlare detected as their origin IP address.
CloudPiercer uses an array of techniques to reveal a targets real IP address. They estimate that 70% of sites protected by cloud WAF providers have their origin IP address exposed. More detailed stats and our own data is included below.
Keeping the Origin IP Address Secret is Difficult
There are many ways to discover a site’s origin IP address. You can:
- Look up the IP address of subdomains like mail.example.com and ssh.example.com. In many cases they point to the origin IP address. (See below for detailed statistics)
- Use an IP history database like viewdns.info to look up where the origin IP was hosted before the website started using a cloud WAF.
- Use the site certificate and a search engine like Censys.io or Shodan.io to locate the origin IP address where the certificate is installed.
- Perform an action that gets the site to connect somewhere, revealing its origin IP address. For example, on WordPress you can initiate a pingback to a site which will cause it to connect back to you, revealing its origin IP address.
- Use DNS records like SPF which might reveal the origin IP or adjacent addresses.
- Examine site HTML source which may include subdomains that point to the origin IP address.
- Examine public source or log files which may include the origin IP address or subdomains pointing to the origin.
The original designers of the net never intended IP addresses to be secret. There are no standards or conventions on the net that describe how an IP address might stay secret. In fact standards like SSL certificates and de facto standards like WordPress Pingback make it virtually impossible to keep an IP address hosting a website secret.
Search engines like Censys.io and Shodan, well known in the hacker community, go around and index the internet at a network level, indexing SSL certificates and network service information. They provide a treasure trove of information to attackers who are looking for targets.
Censys.io lets you easily search for the origin IP of any website by using that sites certificate. The results below are from a search on Censys using the certificate common name (website hostname) of Reddit.com which is behind Fastly and Cloudflare. An attacker would use the origin server addresses to launch an attack that bypasses the cloud WAF.
34% of Cloudflare sites can be bypassed using just one of these techniques
To determine the scale of this problem we did an automated survey of 30,753 randomly selected Cloudflare customers. We tested how many sites are accessible by simply bypassing Cloudflare and accessing the site directly. We only used one technique to discover the origin IP: Looking up a common subdomain. We didn’t use the pingback technique, site certificates or any of the several other techniques detailed above.
We found that we could directly access 34% of Cloudflare customers, or one in every three customers, bypassing any security that Cloudflare provides, by simply looking up a common subdomain.
To perform the audit on subdomains, we took the following steps:
- Verify the customer is using Cloudflare and store a copy of their website home page. We checked HTTP and HTTPS and the site with and without the www. prefix to locate it.
- Look up common subdomain IP addresses.
- Verify the subdomain IPs don’t point to Cloudflare.
- Remove duplicate IPs.
- Try to fetch the site from the origin IPs directly by connecting directly and specifying a ‘Host’ header in the HTTP request.
- Where we received a response, compare the original page title and first asset (first item in a src= attribute) with the page we just fetched directly.
- If they match, then count it as a successful bypass.
As you can see from the chart below, looking up the IP address using the ‘mail’ subdomain is by far the most successful technique. Over 50% of cloud WAF customers that reveal their origin IP do so through the ‘mail’ subdomain. The ‘ssh’ domain is also a common culprit, likely because Cloudflare’s own documentation use it to illustrate how you can connect directly to your own origin IP address.
The Fix: Protect the Endpoint and Prevent Bypass
Endpoint protection has taken the industry by storm during the past year. Almost every major vendor is providing some form of endpoint security. So what is the endpoint exactly? An endpoint can be defined simply as: the final target a hacker is after. In the case of desktop security, it is the workstation a user works on. In the case of mobile platforms, it’s the smartphone a user has in their pocket.
In the case of WordPress, the endpoint is the actual WordPress installation that the attacker is trying to compromise. We believe that to best protect a website, you need to protect the endpoint.
The first benefit of protecting a website at the endpoint is there is no way for an attacker to bypass the security mechanisms. They can’t go around the Wordfence Firewall because it is an integral part of the endpoint. To use the endpoint application, you have to interact with our firewall.
The second major benefit is that, by protecting the endpoint, we can provide a defense in depth strategy. We don’t just provide a firewall. We also include a malware scanner and a range of other features. It’s not feasible to include a malware scan unless you are executing on the endpoint itself.
Conclusion
To fully protect your investment you need to employ an endpoint strategy that takes a defense in depth approach to security. Wordfence takes this approach.
As our company has evolved we have had to consider whether we would invest in “cloud” security or focus on protecting our customers where their assets are. We chose to stay on the endpoint and the industry has now also shifted their focus to protecting the endpoint.
In our opinion, using a cloud WAF is like hiring a security firm in Los Angeles and asking all of your visitors to go through their Los Angeles offices before visiting you in New York. We believe in posting guards where the assets are and putting additional defenses behind that first layer of security. This proven approach works for the over 1.5 million sites we protect and it is where the industry is headed.
Comments
10:21 am
I had no idea. Thank you for not only enlightening us, but also protecting us and our endpoint assets!
10:36 am
Amazing post! I really learned a lot. Wordfence is really leading edge and I appreciate these blogs.
Security is so important, and seemingly impossible. Thanks for explaining the 2. I never really understood "the cloud" before.
I am still a a little fuzzy on IP addresses. Isn't that more like a zip code, than one address, for one website?
What did we do before you came along? Thanks, Mary
11:31 am
Hi Mary,
An IP address identifies the 'server' that hosts your website. You can have one or many websites at that IP address. In the case of the large WordPress hosting providers, it's usually many websites per IP address.
Think of your IP address as a kind of shared telephone number that visitors use to visit your website using a web browser. Once they make the phone call, they need to tell the operator which website they want. They do this by sending what is called a 'Host' header which contains your website name.
So when we bypass Cloudflare or another cloud firewall, we call up the website IP directly, then once connected, we tell the operator (or web server in this case) that we want to visit example.com and it gives us that website. That's how a hacker accesses a website directly when they want to bypass a cloud firewall. Once they're connected to the IP and have told the server which site they want using the 'Host' header, they can send an attack payload directly to the website they want to attack.
And so what you need is someone behind the operator who is a security guard. That's where Wordfence comes in. We do our work within the website code itself to block attacks once the attacker gets past the 'operator' and we also continually scan your code for intrusions.
Hope that makes sense.
Mark.
8:45 pm
I like the analogy and appreciate the explanation. :)
10:40 am
Here, Here! I absolutely agree! I have Securi, and I have Wordfence. I consider both essential. Securi, my cloud firewall is sufficient for the general audience, and an unaware newbie hacker. The web firewall I do not consider as being sufficient for a hacker that knows how to get around the web firewall, as you have previously described. I consider it mandatory to have the home site protected. I relate this to an item that happened in real life. A lady in a nearby town paid good money to have two K-9 trained German Shepards trained for home protection. She put them in her back yard. She considered them as her protection wall against people coming into her house. A home invasion happened, with the thieves kicking in the FRONT door. The dogs were in the back, locked out of the house, and were no help. She was robbed. Her 'wall', the dogs, had been by-passed, and were useless. For protection, it is best to consider all avenues of approach, not just one approach vector. Multi-factored duplicity is best, providing that the over-all system functionality is not affected. I have found that Wordfence has picked up folks that by-passed the cloud firewall, and attempted to access my system. I appreciate the hardening of my system that Wordfence provides, at the doorway to my system.
11:45 pm
I like your analogy to the woman with the German Shepherd!
However, I'm questioning the need/value for a cloud firewall like Securi or CloudFlare. Is endpoint security sufficient, or is there really an additional need for a cloud firewall?
10:48 am
Interesting, how do you deal with DDoS attacks with your endpoint security then ? How many of those shared hosting providers are prepared for a large scale attack and wont just throw you off the bridge and tell you to find another hosting provider if your website gets attacked ?
At least with a cloud solution you have more control, you can block all requests coming from IP addresses that are not your firewall and solve your problem.
If you are determined you can certainly solve those issues, you can have the SSL installed on the firewall so your real IP is not visible, for mail you can use a 3rd part provider and not your hosting server to handle mail. All those DNS history websites rely on old data, just move to a new IP after setup the cloud firewall and they wont know where you moved.
If CloudFlare is not doing its job then why do they have so many users ? Why are there tools to find a way to bypass it ? Because its effective and hackers don't like that.
11:13 am
Hi there,
Many hosting providers have their own DDoS mitigation. If an attacker knows your origin IP address then they'll just DDoS you directly, bypassing the cloud firewall.
You would need an endpoint firewall to do that.
Installing SSL on a cloud firewall does not hide your IP. Installing SSL on the endpoint actually exposes your real IP even further by making the certificate available to crawlers like Censys.io and Shodan.io which index your certificate which includes your website name. Attackers can then just do a search and find the origin IP you're using.
And that's the real problem we're describing. You end up in a game of hide the IP address and the Net was not designed to do that. You can keep running, but as Censys and Shodan index your SSL certificate which contains your hostname, they'll keep unearthing your origin IP.
Because they raised $180 million in funding which buys a lot of marketing dollars.
Because researchers are interested in providing real security rather than just a feeling of security.
11:22 am
Thanks Mark for your detailed response to Johnny. Clears the air.
10:49 am
Thanks to all of you for your hard work!
10:54 am
Yes, requests can bypass any WAF that filters incoming traffic when the WAF is configured by a DNS change and the attacker knows the origin IP address. You forgot to mention though that if the origin server limits incoming traffic to the IP addresses of the WAF as that is the only place legitimate requests will come from as the WAF filters all incoming traffic, it would prevent this as any request that bypassed the WAF would get a 403 forbidden error or timeout based on what solution was implemented to restrict access to only the WAF IP addresses.
If it is using Apache you can use the .htaccess file to do that or with Nginx you can use the Nginx configuration file. Another option is to use a firewall on the origin server to limit traffic on HTTP (port 80) and HTTPS (port 443) to only the IP addresses of the WAF.
11:22 am
Hi John.
Agreed. However most people aren't aware of this and as our audit shows and don't have it enabled. Our research demonstrates that one three Cloudflare customers have their origin IP exposed and don't have this configured. According to CloudPiercer that number is much higher.
I'd also add as a footnote: The cloud waf providers big claim is they will protect you against DDoS attacks. As long as your origin IP is exposed, they can't protect you against DDoS because an attacker will just DDoS the origin. Furthermore, no amount of .htaccess rules or even OS level 'iptables' rules will protect you from DDoS because once your server's local network is overwhelmed with traffic, your site is down.
You also don't get a layered approach to security if you have a cloud WAF because you only have firewall rules with no local malware detection.
So what we're advocating for here is that you use endpoint security which has a layered approach to securing your site. And then use a hosting provider that includes DDoS mitigation.
Mark.
11:54 am
Hi Mark,
You make a great point, if it is something "most people aren't aware of" which is reasonable and I believe is true, isn't it the job of people in the security industry to educate those people who don't know of that option (and provide that as a possible solution)?
Regarding DDOS attacks, you see a lot more layer 7 attacks targeting the application layer than layer 3/4 targeting the network layer, which if you implemented a cloud WAF and impelmented the steps I mentioned to only allow that WAF's IP addresses send requests to your origin server, it would help stop that.
In regards to layer 3/4 attacks targeting the origin server and if the origin IP address is known, regardless if a cloud WAF or a solution that is implemented on the application in this case WordPress such as WordFence, will either solution be able to stop that? No, so that would be left in the hands of the hosting provider to mitigate and most hosting providers don't have the capability to even mitigate a 20 GBPS DDOS targeting layer 3/4, which is a low number and more realistic common size attack compared to the large amounts you often heard quoted in the media of 500 GBPS etc.
Therefore a cloud WAF, only allowing the WAF's IP addresses on the origin server, and taking steps to hide the origin server IP address is an effective solution to layer 3/4 attacks as well as it makes the origin server IP address harder to find. If you use a solution on the application itself, the hosting address will always be known.
Yes, a layered approach with security is always good, but a cloud WAF's purpose is not monitoring or detection and I think that is well known. That is why some WAF providers offer a monitoring solution (external monitoring and/or monitoring scanning the files on the origin server directly) and if you only use a cloud WAF, you can still implement a solution on the origin server for monitoring, if you use a VPS/dedicated server, TripWire and OSSEC are two examples. There are also many monitoring solutions you can implement on the application itself such as WordPress or other content management systems as well.
Thanks,
John
12:25 pm
We're going to have to agree to disagree on that. You're assuming an attacker is targeting the hostname of the target. I'm assuming they're smarter than that and will target the origin IP.
You'd be surprised. DDoS mitigation has come a long way. We've gone beyond devices that automatically black-hole the target IP taking it offline. Instead, now we'll have an upstream provider route the traffic through a layer 7 DDoS mitigation filter. Wordfence uses a technique like this [Edit: To protect our own infrastructure, to be clear] in cooperation with an upstream tier 1 provider and it's incredibly effective while keeping our services online. Hosting providers are now buying that same service from their own upstream providers - and it's quite cost effective too.
Mitigating DDoS using cloud services with the origin exposed is a naive approach that assumes an unsophisticated attacker. And our thesis is that you can't rely on the origin remaining unexposed. It's a security through obscurity approach which we all know is a really bad idea. So instead, focus on protecting the endpoint both with DDoS mitigation and with an endpoint firewall and other tools.
11:04 am
I appreciate your very informative articles and the fact Wordfence is clearly designed to thwart the endpoint attacks. I use Sucuri on one of my Wordpress sites (paid option). So you are saying an attacker can easily bypass that security and potentially compromise my site despite the fact I have one of the best security options available (or so I'm told) installed and working 24/7. Hmmmm.......
4:23 pm
Hi @terrance
If you're blocking direct access to the server via the rules we provide in your dashboard you'll mitigate the bypass described in this article.
If you have specific questions on what you're covered with feel free to reach out to us directly. We'll gladly address any concerns.
Tony
11:22 am
To be clear, very few people have a .htaccess rule configured to only allow WAF access - as the research above shows.
Secondly, cloud waf's claim to protect you against DDoS. That's one of their big selling points. A .htaccess rule only allowing the cloud WAF IP's does not fixed the risk of a DDoS attack targeting your origin IP. If they DDoS the origin, you're still down, htaccess rule or not. We don't protect against DDoS, but neither do cloud WAF's in this instance.
Thirdly, be very clear that the cloud WAF provider - Sucuri in this case (Tony is their CEO) has absolutely no visibility on attacks that go around the cloud WAF and target you directly. So if you have a misconfigured .htaccess file, or they discover some other attack method that can bypass .htaccess, the cloud WAF can't protect you against it AND they don't even know it's happening because they don't see and log that traffic.
As I've mentioned in another comment on this thread, I really think the term cloud WAF is misleading and the business should not exist in the first place. Public servers on the Internet do not a firewall make.
Mark.
11:18 am
You forgot the classic security fail of using your own "protected" server to send out transactional emails like password resets etc.
This will reveal your server ip immediately.
11:23 am
Exactly. Thanks Mike - we were so busy thinking of pingbacks that I forgot that the mail header would include your origin server's IP address.
1:23 pm
I choose for the Nuclear Option on both my WP site and Cloudflare and choose to disable the possibility to send/receive email completely.
Cloudflare simply can not handle email over their secured network - it will always reveal the original IP
https://support.cloudflare.com/hc/en-us/articles/200168536-Why-do-I-have-a-dc-subdomain-
The safest thing to do is to delete your MX record and the dcxxxxxx subdomain (former direct-connect)
In WP the trick was to mis configure my mail settings so that if WP tries to send an email it will result in an error.
In my case, I really don't need email, but for many sites, the situation is completely different of course. I guess the best solution would be a separate mail server.
PS: you can cheat hackers playing with your CF settings. Till a couple of years ago I directed in CF's dashboard my 'direct-connect' to an non existent IP address in a country far, far away. Crimeflare still believes I'm there ;-)
11:55 am
This was a great article to read and informative. It really hits upon some great points about security, and why using Wordfence is essential to WordPress endpoint security. I also enjoyed the graphics in the article, nice touch!
Mark @ Zunamic
1:21 pm
Unintended Consequences
2016 brought an onslaught of new spam from globally distributed IP ranges – the mail host are certainly responsible for allowing the spam through their networks and should be held accountable.
The recent most disturbing trend I see is huge batches of emails from multiple mail servers from multiple countries – all with the same html basic template and layout all linking to websites hosted on CloudFlare servers.
This is a huge cost savings to spammers because you cannot report abuse to cloudflare based on the IP lookup of the domain name in the link. Cheap email services are easy to find and swap between when the IP ranges get blacklisted - but not having to take down the site linked to in the spam is a huge win for the spammers.
WordFence continues to be my best friend in WordPress = and I appreciate you and the team for all of your hard work!
2:13 pm
However endpoint protection can't protect effectively against a DDoS attack whereas Frontend Protection cloud bases services can (when properly configured and when the direct address is hidden).
5:33 pm
Fascinating article. Thank you - so glad I read it.
There I was, smug in my belief that we had joined the ranks of the privileged and relax knowing we were totally protected with Sucuri (as a paid subscriber).
It is clear this is not the case, even if we have have modified our htaccess file to force people to access the site only through the cloud firewall.
I have reinstalled Wordfence based on the article!
So my question is: is the best scenario to have a cloud firewall such as Sucuri AND Wordfence?? Or would you go so far as to say Sucuri is redundant?
AND also, is it ideal to have the Wordlfence premium version. We used to but I see it has gone from USD39 per site to USD99 which is a big jump.
Extremely keen to get your response Mark on both counts.
Many thanks.
6:44 pm
Wordfence's firewall is the best option for WordPress by a significant margin if you are hoping to stop a hack. If you want DDoS protection, talk to your hosting provider about your options. If an attacker is able to discover your origin IP address, and as we've demonstrated in this article it's quite hard to prevent this from happening, a cloud WAF will not protect you. So the best approach is to protect your origin from DDoS by choosing a hosting provider that has this kind of protection.
So yes, I'd say I don't really understand what purpose a cloud WAF serves other than to act as a content distribution network the way Amazon's Cloudfront, Akamai and Cloudflare do.
Ironically, if a service like Cloudflare wants to provide a content distribution network, they need to place their servers as close as possible to the client (the web browser). That means, to be a good CDN, they need to be geographically as far away as possible from the origin server they're trying to protect, if they're also trying to be a WAF.
I'd also point out that for a cloud WAF to do their job, you need to provide them with your private SSL key so that they can decrypt your SSL traffic on their server, inspect it and decide if they want to forward it or not. Because they also try to be a CDN, they have 'edge servers' all over the Net including, in Cloudflare's case, places like China. Your private key needs to live on all those servers which increases risk. There is a way to keep the private key on your own machine and have the edge servers make a kind of callback to the origin, but that adds latency. So again, the conflict between being a CDN and being a WAF.
I don't actually think the cloud WAF business should exist and the industry is clearly moving in that direction. What I think will eventually happen is the current cloud WAF providers will just become CDN's with analytics thrown in and basic security. The heavy lifting, as is the case with enterprise firewalls now, will be done way closer to the network that is being protected.
Mark.
11:51 am
I agree that Cloud WAF should focus more on CDN and DDOS protection, that's what they were designed for, that's what they are good at.
2:19 pm
Hi Mark,
Thank you for your detailed response. I still feel that (and many of the comments on this post endorse this) the best solution is to have Cloud WAF as well as Wordfence. Maybe this is something you could discuss please as many folk may be be confused and thinking that they should just dispense with CLoud WAF (although that indeed is what I do glean from your comments).
Another question I have relates to WF's use of site resources. Would you be able to provide an indication as to what extra load can be expected and whether resources such as memory might have to be upgraded to accommodate this.
And finally, do you see a problem with using another security in conjunction with WF (such as iThemes) - just for a bit of extra beefing up?
I'm all set to go premium - I just want to be sure first.
Many thanks!
Bryan
3:11 pm
Hi Bryan,
Wordfence doesn't use very much additional resources at all. Even the smallest hosting accounts seem to have no problem. You can absolutely run a cloud WAF in front of us. I in fact we have built in support for Cloudflare and they way they hand us the attacker's real IP address. Regarding installing another security plugin - sure you can do that. We haven't had any reports of problems. But it feels like overkill to me.
Regards,
Mark.
3:55 am
This is great!
I believe Sucuri has requested this as well, but it's been denied. Maybe there was a misunderstanding. Now that you're supporting Cloud Firewalls, what needs to be done for Sucuri to get the same support you've built in for CloudFlare?
Tony
10:30 am
Hi Tony! Actually this data comes in from the CF firewall as an HTTP header, so we just read the header.
Just reach out to me and let me know what you guys need. We're already whitelisting your IP's AFAIK. Let me know what else you need.
Mark.
10:21 pm
Nice article...
The end-point needs to be very secure that is for sure, yet any Cyber Security needs multiple layers, as you don't want to rely on one thing to guard your website/server. I use CloudFlare, then WordFence Premium for the WAF and also use the iTheme Security (Free Version) for some extra security, in addition to server security...
...and most importantly MindSet, Security needs to be a ‘mindset’ as hacker access to your website/server can start with you...
1:19 pm
Great write up. We deploy concentric rings of ever tightening security on our network and our servers, including both WAF, monitoring, and automatic countermeasures, but we consider Wordfence the application level security service that rounds out the process of protection. It's not just peace of mind for our customers, it's peace of mind for us. And it's one more weapon in our arsenal of providing the best security we can offer for our customers. Thanks again for a great product and service. We highly recommend Wordfence.