Cloudflare Data Leak: How to Secure Your Site
Cloudflare has experienced a data leak over a 5 month period that mixed sensitive data between websites and visitors. A visitor to one website using Cloudflare may have seen data from another website using Cloudflare that was being sent to a completely different site visitor.
Some of the leaked data has been indexed by search engines who have been working over the past few days to try and remove the data from their caches.
In this post I am going to explain in simple terms, what occurred and what you need to do about it.
If you are a WordPress user and simply want to know how to secure your site, you can skip to the What Should I Do section below. I have included some information for non-WordPress site owners in that section too.
What Happened in the Cloudflare Data Leak?
Cloudflare provides a firewall and content distribution service. Their servers are between your website visitors and your own web server.
Under normal circumstances, cloudflare returns the data each site visitor requested to that visitor. This may be public or sometimes private information and it is usually done over a secure channel. Each website visitor only sees the data they requested.
From September 22nd, 2016 until February 18th 2017 (last Saturday), Cloudflares servers in some cases mixed data that belonged to one visitor to a website, with data belonging to another visitor that was visiting a completely different website.
The worst data leakage occurred between the dates of February 13th and February 18th when one in every 3.3 million requests to Cloudflare’s servers was leaked.
During the period when the leak occurred, visitors to certain websites would see what appeared to be garbage data mixed into the web page they were viewing. That garbage data was data from a memory leak. The data was in some cases sensitive and included security tokens and other sensitive information.
According to Tavis Ormandy, the researcher who discovered this data leak:
“The examples we’re finding are so bad, I cancelled some weekend plans to go into the office on Sunday to help build some tools to cleanup. I’ve informed cloudflare what I’m working on. I’m finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We’re talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.”
Data leakage occurred when a site visitor or search engine visited one of 3438 domains hosted behind Cloudflare’s servers, according to Cloudflare’s CTO who posted a comment on Hacker News. However, any of Cloudflare’s customer websites could have had their response data mixed into data returning from those 3438 websites. Tavis Ormandy confirms this in the same Hacker News thread.
You can see an illustration of this data leakage in the diagram below. Any visitor to an ‘affected website’ which is one of the 3438 websites, could have had data from any one of over 5 million Cloudflare customer sites mixed into their response. Website 1 and Website 2 which are not ‘affected’ websites could have experienced data leakage to visitors of the ‘affected’ website.
What Data Was Leaked?
The data that was leaked could include passwords, cookies and authentication tokens. If an attacker is able to access the text of your cookies, they may be able to use them to sign into your website.
Internal Cloudflare private keys used to secure data being transferred between Cloudflare machines were also leaked.
At this point one should assume that there is a small chance that any private data transferred from any Cloudflare customer website to a site visitor may have been leaked between September 2016 and February of this year.
According to Cloudflare, no private SSL customer keys were leaked from the memory of their servers. A private SSL key is a key used to secure visitor connections to your website. If the your private SSL key is leaked, an attacker could listen in on all traffic to and from your website.
Has the Leaked Data Been Stored Somewhere?
Unfortunately Google and other search engines have been crawling the web during the time that this leak was occurring on Cloudflare’s systems. Those search engines stored the leaked data when they indexed one of the 3438 affected websites.
When viewing cached pages in Google, it is still possible at the time of writing (7pm Pacific Time on Feb 23rd) to view cached sensitive data in Google’s search results.
Similarly it is possible to view sensitive Cloudflare data in DuckDuckGo’s search results.
Since this leak was discovered, Google and other search engines have been working to try and remove the sensitive data from their caches. Based on what we are seeing this evening, there is still some data that needs to be removed.
What Should I Do if I use Cloudflare on my Website?
According to a conversation on Hacker News between the Cloudflare CTO and Tavis Ormandy, the security researcher who discovered this, any customer of Cloudflare’s could have been affected by this data leak.
WordPress site owners: Change your wp-config.php salts. This will log everyone out and invalidate cookies and sessions
If you are using WordPress, we recommend you edit your wp-config.php file and change all of the ‘salts’. This will automatically log all of your users out. This protects you and your site members in case any of their cookies have been stolen. Once you make this change, an attacker will no longer be able to use stolen session cookies from your site to sign in.
You need to change the following section in your wp-config.php and save it:
We suggest that you change the highlighted text in your wp-config.php to a long random string of characters and numbers. You can also use the link in the comment above the ‘define’ statements to generate a salt.
Non-WordPress Site Owners: Invalidate Sessions
If you use a different publishing platform, you will need to ensure that all sessions are invalidated. That means that your site visitor login cookies need to be made invalid. You will need to consult the documentation of your particular publishing platform to determine how to do this.
Suggest your site members change their passwords and change your Admin passwords
As a precautionary measure, you should suggest that your site members change their passwords. You should also change any admin level passwords.
You may need to comply with any data breach reporting requirements you have
This bug in Cloudflare’s systems is being described as a “data leak”. It is unclear at this point whether it is considered a “data breach”. A private database of customer personally identifiable information was not stolen. However, private data that may have included customer PII was leaked, no matter how small.
If you have HIPAA, PCI or other reporting requirements that relate to data breaches, you may want to get advice on whether you are required to report this incident.
Check the Search Results
It is difficult to determine if any private information from your site has been stored in the search results. However, we recommend that as a precautionary measure you do a few Google searches with your domain name in quotes. Add the following text to the search:
-site:example.com
Replace example.com with your own site domain. Take note of the minus sign before the word ‘site’ above. This will exclude results from your own website. You can exclude results from other sites using the same operator several times. You can also try adding the following in quotes:
“CF-Host-Origin-IP:”
If you do find any results, report them immediately to Google for removal.
Who Discovered This and How?
Tavis Ormandy discovered this data leak in Cloudflare’s systems. He is a security researcher employed by Google’s Project Zero. Project Zero is a Google team who works on trying to find zero day (previously unknown) vulnerabilities.
Tavis discovered the data leak while analyzing Google search results. He noticed data that appeared to be a raw memory dump and he and his colleagues took a closer look and discovered it was a data leak in Cloudflare’s servers that was leaking data between websites.
Tavis is a well known researcher who has done ground breaking research in the computer security field over the past few years.
Has This Problem Been Fixed by Cloudflare?
The Cloudflare team have fixed the data leak on February 18th which was last Saturday. You can find a detailed technical post on Cloudflare’s blog describing what caused the leak and how it was fixed.
According to Techcrunch, Cloudflare have not notified customers like Uber and OkCupid directly.
Where Else Can I Read About This?
The following are the most authoritative resources discussing this issue:
- The thread on the Project Zero mailing list where Tavis discloses the issue.
- The Cloudflare blog. This is a highly technical post describing how they fixed the issue.
- The Hacker News thread that first brought public attention to this issue.
How can I help?
Please share this information with any other site owners you know to help them secure their websites. Specifically, if you know any WordPress site owners you should suggest that they invalidate their user sessions using the instructions above.
As always I will be around to respond to any comments or questions you post below.
Comments
11:43 pm
Good post! thanks for keeping us updated.
You only need to change salts when you used cloudflare correct?
7:07 am
Yes that is correct.
11:47 pm
Thanks for the informative post. I changed the salts as suggested but now I'm getting this error on the cloudflare plugin "csrf token not found it's possible another plugin is altering requests sent by cloudflare" whenever I try to change anything there. Before changing the salts it ran perfectly fine.
Any help would be appreciated. I love how you guys work so hard to keep us updated.
11:50 pm
Mark, in brief, the WordPress site admins/owners should:
1) change the 'salts' in wp-config.php
2) change the admin password
3) ask users/members (if any) to change their passwords
4) do a Google search for any information leak and report to Google (optional)
Is that all?
Thanks
7:08 am
Yes that is a good checklist.
11:54 pm
I had a lot of difficulty finding salts in my CPanel files. Could you be more specific as to how to get there?
7:08 am
You can find them in a file called wp-config.php in the base of your WordPress installation directory. You will need to use a file manager or FTP to access that file.
2:03 am
Just wanted to say thanks for this article, the email you guys sent out was the first I heard of it.
The one website my company has running through Cloudflare isn't Wordpress and thankfully wasn't effected, but having the heads up is greatly appreciated!
7:10 am
You're welcome Mike. We wrote that article as fast as possible without compromising on quality. So a few news outlets published before we did but several of them have the story wrong. e.g. Some of them are reporting that only 3438 sites leaked data. That's incorrect. In fact if you accessed an of those sites, you would SEE leaked data. But the data could have leaked from ANY cloudflare customer.
Mark.
2:13 am
Hi,
I'd like to add this very thorough advisory to my announcements page. Can I reuse it and if so how would you like me to post attribution?
Thanks.
7:10 am
Just include a link to us as the source. Thanks Martin.
3:51 am
Well, if I understood it correctly, the first causes of the bug (which is indeed a proxy parser error) are HTML tags not properly closed. So the first thing to do should be to check its own websites syntax :-)
7:12 am
It doesn't matter if your own website uses perfect HTML syntax. Your data could have been leaked to a visitor or crawler who was visiting a site that had the syntax that triggered the bug. About 3438 sites had that bad syntax.
This has been misreported by some media outlets to give the impression that only 3438 sites leaked data. That's incorrect. In fact 3438 sites triggered the data leak. Any cloudflare customer may have had data leaked to a visitor or crawlers accessing those 3438 sites.
Mark.
4:46 am
1. Is this something we should be concerned about if we have a sitewide SSL?
2. Does this only affect Cloudflare users?
3. Are you planning to add a feature to Wordfence to address this?
7:13 am
SSL doesn't protect you because Cloudflare decrypts your traffic on their servers. That is one of the things I dislike about Cloud WAFs: The fact that they have to decrypt your traffic to be able to inspect it.
Yes this only affects websites using Cloudflare. HOWEVER, if your website does not use cloudflare, but you use a REST API or service that does use cloudflare, then you should consider anything coming back from that API as potentially leaked. I don't think this is a common case.
Mark.
8:01 am
Mark, do you recommend a service that, in your opinion, does a better job than Cloudflare?
8:10 am
Sure. Use a endpoint firewall like Wordfence to protect your site. If you want a CDN use a CDN that does not provide a firewall.
Avoid cloud WAFs. They have a range of issues I'll detail in a future post. But for now: They don't even know if a user is signed into your site or what permission level they have, so they can't use that in their rules and decision making. The reason they don't know is because they live off-site.
Mark.
9:46 am
"They don't even know if a user is signed into your site or what permission level they have, so they can't use that in their rules and decision making."
A little bit like letting someone into your living room and *then* determining whether they're a burglar?
2:01 pm
Is there a CDN you would recommend, or in your opinion, do you think it's better to just avoid using one?
3:13 pm
CDN's are great. It's cloud WAFs that I would avoid. If you use a CDN, don't use the cloud WAF functionality. Also don't let it decrypt your SSL traffic. It is breaking end-to-end encryption.
Mark.
5:04 am
Interesting. I've had a lot of pressure to use Cloudflare, resisted as it always seemed to over-promise, super lame they let this happen. I hope heads will roll.
7:14 am
They should. And you're right to be relieved. This is a serious pain for any site that uses them.
6:13 am
my website was affected by this issue thanks to wordfence to give us the recent news l change all the authentication key
7:14 am
Mark, thanks so much for the heads up on this. I wanted to comment to congratulate you once again on providing such a great service to the web at large. As a Cloudflare customer, I received a notice from them as well to inform us about the bug and its effects. YOUR email came in 6 hours before theirs.
Your proactive, responsible approach to security exploits is a godsend for the community of web developers at large (not just WordPress users).
Thanks & keep up the great work.
Robbie Ferguson
7:43 am
I have a sense you may have some competitive interest in presenting Clodflare in certain ways...? This is not the first time I have had this sense... And at the same time I deeply appreciate being able to use both Wordfence and Cloudflare in their free versions, and the strong integrity of the work both organizations do for the broader internet and Wordpress communities, and how that serves websites that are out there for other than commercial or monetary purposes...
Here is another perspective on this...
Dear Cloudflare Customer:
Thursday afternoon, we published a blog post describing a memory leak caused by a serious bug that impacted Cloudflare's systems. If you haven't yet, I encourage you to read that post on the bug:
https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/
While we resolved the bug within hours of it being reported to us, there was an ongoing risk that some of our customers' sensitive information could still be available through third party caches, such as the Google search cache.
Over the last week, we've worked with these caches to discover what customers may have had sensitive information exposed and ensure that the caches are purged. We waited to disclose the bug publicly until after these caches could be cleared in order to mitigate the ability of malicious individuals to exploit any exposed data.
In our review of these third party caches, we discovered data that had been exposed from approximately 150 of Cloudflare's customers across our Free, Pro, Business, and Enterprise plans. We have reached out to these customers directly to provide them with a copy of the data that was exposed, help them understand its impact, and help them mitigate that impact.
Fortunately, your domain is not one of the domains where we have discovered exposed data in any third party caches. The bug has been patched so it is no longer leaking data. However, we continue to work with these caches to review their records and help them purge any exposed data we find. If we discover any data leaked about your domains during this search, we will reach out to you directly and provide you full details of what we have found.
To date, we have yet to find any instance of the bug being exploited, but we recommend if you are concerned that you invalidate and reissue any persistent secrets, such as long lived session identifiers, tokens or keys. Due to the nature of the bug, customer SSL keys were not exposed and do not need to be rotated.
Again, if we discover new information that impacts you, we will reach out to you directly. In the meantime, if you have any questions or concerns, please don’t hesitate to reach out.
Matthew Prince
Cloudflare, Inc.
Co-founder and CEO
8:08 am
Hi John, [In reply to John Fridinger the poster. Not Matthew Prince. This was not posted on the blog by Cloudflare CEO, to be clear. It's a repost from a user.]
I have so many objections to the way CF has handled this it's tough to know where to start.
1. The CEO just emailed you to let you know about a "memory leak caused by a serious bug that impacted Cloudflare's systems".
Shouldn't he be telling you about a data leak that has impacted you and your users?
2. The blog post has over 2000 words of some of the most technical copy I've seen for a while. It is completely inaccessible to normal users. Then, after 2000 words of tech jargon, they include this:
Quote: "More concerning was that fact that chunks of in-flight HTTP requests for Cloudflare customers were present in the dumped memory. That meant that information that should have been private could be disclosed.
This included HTTP headers, chunks of POST data (perhaps containing passwords), JSON for API calls, URI parameters, cookies and other sensitive information used for authentication (such as API keys and OAuth tokens).
Because Cloudflare operates a large, shared infrastructure an HTTP request to a Cloudflare web site that was vulnerable to this problem could reveal information about an unrelated other Cloudflare site.
An additional problem was that Google (and other search engines) had cached some of the leaked memory through their normal crawling and caching processes. We wanted to ensure that this memory was scrubbed from search engine caches before the public disclosure of the problem so that third-parties would not be able to go hunting for sensitive information." END QUOTE
It's a jaw dropping disclosure and should have been at the top of the post.
3. The title of the blog post is: "Incident report on memory leak caused by Cloudflare parser bug"
Why not just title it "Move along. Nothing to see here."?
This is a major data leak (a breach really) that affects over 4 million websites and probably over half a billion website visitors. It was in production for months. And is now world-wide news.
4. They have not made it clear that any Cloudflare customer may have had data leaked. The impression some major media outlets have been left with is that only 3438 websites had data leaked. No. If you accessed any of those 3438 websites you may have seen data from ANY cloudflare customer.
5. The email from their CEO was only sent this morning after every tech publication in the world had already picked up the story. They're not exactly getting out in front of this issue. They're reacting to it and doing the minimum.
I could go on.
Mark.
8:20 am
As a DIY non-tech person, how do I tell if my WordPress site uses Cloudflare? I can't find it in the plugins or in the WordPress dashboard.
Is this an issue if I am a premium Wordfence subscriber?
Thanks!
8:55 am
Hi John,
You can use this site: http://www.doesitusecloudflare.com/.
Yes it is an issue even if you're a Wordfence Premium customer. If you've used Cloudflare, they decrypt your traffic on their servers and there's nothing we can do to prevent that. That traffic may have been leaked. Sorry.
Mark.
9:05 am
Hi Mark,
I second you, CF has shied away from their responsibilities. The just tried to make this whole incident appears minor. Though I have been using both cloudflare & Wordfence both, I find your post more direct & educating on post measures for the affected customers than CF.
Thank you for keep doing the Good work.
Amar
10:02 am
Yet ANOTHER reason to avoid CloudFlare. That list of run far far away just keeps growing by the year.
I wouldn't use CF if they gave my company access to their premium service for free for life. No thanks!
4:02 pm
Thanks for the explanation. For those of us who use Cloudflare as part of our SiteGround hosting service, do we need to make the changes you suggest, or is this something SiteGround would be taking care of? Thanks again.
4:35 pm
I have not heard of any hosting companies making this change. I strongly recommend, no matter which hosting company you are using, you make this change yourself.
Mark.
6:36 pm
Thanks for the info. I also changed passwords on anything that could take down my websites if accessed:
1) Wordpress Salts
2) Wordpress password
3) Cloudflare password. Then purge everything from cache.
4) Cpanel password
5) Namecheap password
6) Hosting company password
Time for a password app. The breaches and software bugs are out of control!
6:45 pm
Does this apply to sites that only use cloud flare for dns? Ie they don't have the cloud icon turned orange. They just use for DNS lookup.
7:45 pm
How to change the salt key?
Btw, My site performance was good. But from several days my site loading speed above 6 seconds. I also contacted my hosting provider, but they did from their end. But the problem still exists.
Should I switch to another CDN provider?
Jyoti,
10:54 pm
Although I am not consciously a client of Cloudflare, the nature of Cloudflare's service to protect against DDOS attacks, makes me concerned that website hosting services "subcontract" Cloudflare at their datacentres that might host websites that might have their data leaked.
Is there any grounding to my suspicions.?
5:42 am
Kudos to Mark for being willing to call out his competitors, time honored technique of marketing positioning, and if honest, is good for everyone. No need to play nicy nice with something as incredibly bad as what Clodflare allowed to happen. They're clearly bad, and need to be called on it. Decrypting all traffic so it can be firewalled? Patently ridiculous. Amazes me these companies stay in business after stuff like this. I wish it were so easy for the rest of us. But if I'd made a mistake like that as a carpenter, bus driver or nurse, I'd be out of a job and possibly even in jail. What is with this "geek immunity?"
9:54 pm
I'm assuming Cloudfare and others have been working with sites like archive.org as well?
I almost started using Cloudfare, since it's offered by my provider but opted to not. Glad I didn't.
5:53 am
HI, I run a wedding photography business. I dont collect any personal data per say (email address and phone number). Neither do I have any users that sign into my website or have log is etc. Its a pretty basic site really.
However, I did receive the email from cloudflare that John Fardinger talks about above. Im no tech expert but the way that was written made me read it twice Once to try and understand it the other to actually try and believe what I was reading.
To me it seems a deliberate way of trying to confuse the layman and at the same time dampen down the issue to tech experts.
Reading between the lines im still not sure what the impact to me is but im certainly now installing word fence and looking at other CDN options.
Great article.
Kind Regards
Greg
4:55 am
all my sites were hacked in jan via cloudflare, over 200...wordfence helped a lot...doing security checks on 200 sites is tough enough, wordfence made it a lot easier for me to find vulnerabilities in my clients sites...thanks to wordfence..i can rest easy knowing my customers are safe...i have discontinued the use of cloudflare, all salts and passwords for sql changed, passwords for logins changed and security logins beefed up..and also removed all the remote SQL entries that the hackers added in all my clients cpanel's..
Thanks wordfence your the best...
Steve
4:49 am
Based on what happened to one of my sites that I put on Cloudflare yesterday, they still have problems to resolve. As soon as I added the site to my account, it began to get traffic addressed to a site that has been hacked and is spewing forth spam. --lawncareseo.com--
Fortunately, I have Wordfence on the my site- that was how I discovered the problem, by looking at the live traffic. All of the attempts were blocked by Wordfence. I was able to pinpoint exactly when the traffic began, and it coincided to the very moment that the site was put on Cloudflare.
2:07 pm
Interesting, thanks for sharing. If you'd like to email us details (mark@wordfence.com) it might be interesting to investigate further.
12:18 am
Finally. I hate CF.
9:03 pm
Thank you.
I am planning to use wordfence.