Service Vulnerabilities: 3 Hosting Companies Fix NFS Permissions Problem
In mid-December we updated our Vulnerability Disclosure Policy to include Service Vulnerabilities. A service vulnerability is any issue with a technology service that represents an exploitable security risk for its users. We made this update in response to a growing trend of security issues we’ve been discovering in commercial services, most often WordPress hosting providers.
Earlier this year we started to see multiple similar cases from three service providers with no immediate explanation. We love a good mystery, and solving this one led to the discovery and correction of a critical vulnerability affecting many of the companies’ customers. The companies were:
- Hostway (and its other brands including Gate, Netnation, Domainpeople, and rebranded services for Qwest/Centurylink)
- Momentous Corp. (includes Rebel and Internic.ca)
- Paragon Group (includes Vidahost, TSOHost, Webfaction and others)
A Note on Disclosure And Responsible Vendors
It’s important to note that vulnerabilities are a fact of life in any service, system or software. Finding, confidentially disclosing and fixing vulnerabilities is how industry works with the information security community to improve the products and services we use and keep the public safe. The process that we use is well established and is widely used by organizations that include Google’s “Project Zero” and Cisco’s “Talos” security group.
When vulnerabilities are found and vendors are responsive, you benefit as a customer of those vendors and can know that your vendor reacts quickly to fix security problems and will likely do so long term, keeping you and your data safe.
A disclosure like this is not an opportunity for “vendor shaming” or a witch hunt. All developers who write enough code write vulnerabilities at some point in their career. It is in fact a moment to celebrate responsive vendors and a well handled incident that left customers and the online community safer.
At Wordfence, we are excited when a vendor works closely with us to fix a vulnerability and responsive vendors garner the greatest respect from our engineering team.
Attacks and investigation
As time went by, we had groups of separate customers suffering identical attacks. At first they were locked out of their administrator account and a user from an unknown IP address had logged into it. Then they saw new administrator accounts appearing from nowhere. Later the attackers started to add malware, spam, or defacement messages to the blog posts or site titles.
In each case we had fairly complete forensic information available to us, but it didn’t lead to any answers about how it was happening. We thoroughly analyzed all of the site files, and didn’t find any malware. We set up detailed logging to show us exactly when the exploits were taking place, and then checked site access logs at those times, and still found nothing. Even the less-used avenues of attack like cron jobs and FTP accounts were clean.
Taken together, all signs pointed to the sites’ databases being compromised. If the attackers have direct access to the database, they can change the existing administrator account’s password, add new accounts, or tamper with site content. It would leave no signs in any of the site’s logs, and would not require them to save malware in any files.
But how were they getting access to the databases? On most hosting platforms, you can only reach the database server through a hosting control panel. Web applications like WordPress can reach it as well. But either way, you still need a site’s particular database username and password in order to make changes to that site. In virtually all WordPress sites, that username and password can only be found in one place: the wp-config.php file.
File permissions
With a shared hosting service, companies use one server to host dozens or even hundreds of websites. Each customer gets a user account on the server, and their site’s files belong to that account. Sometimes groups of users are created, and files may belong to a group as well. In short, every file and folder has both a user and a group which are considered the owners.
Server-level file permissions control who can do what to files and folders. In addition to assigned owners, every file and folder has three permissions settings:
– User (what can the owner do?)
– Group (what can the group do?)
– World (what can any user on the server do?)
It’s possible to make dangerous mistakes with permissions. For example, if you set your website directory to be world-writeable, then any other user on the server can create a file in it. Or, if your website directory and wp-config.php file are both world-readable, then any other user can read the contents of wp-config.php (which include your site’s database username and password). On most servers, PHP scripts run as the customer’s user account, so if a script belonging to user ‘bob’ tries to access a file, it will only succeed if that file’s permissions allow bob (or any of bob’s groups) to do so.
An important note: imagine a folder named home. If your account has “read” permission on it, then you can list the contents of it. If you don’t have “read” permission but you do have “traverse” permission, then you can work with any file or subfolder within home as long as you know their names and have appropriate permissions on them. For example, the server will not tell you that home contains a folder named public_html. But if you already know that public_html exists and you have access to it, then you can still work with it as long as you have “traverse” permission on home.
Network file systems
Managing shared hosting servers can be tricky. Years ago, the hosting company would set up a server with let’s say 500 GB of available disk space. They might choose to limit each customer to 1 GB of usage, and then they could only have 500 customers on that server. But many customers would use only a fraction of the space, resulting in wasted resources. If customers were simply allowed more disk space, the server’s disk might fill up.
Hosting companies have found several ways around this problem. One of them is to use a network file system (NFS) to house customers’ website files. In this setup, an internal group of servers works together to provide disk space, which is shared through the private network with the public-facing servers. So the server that runs your website thinks that it has a massive disk plugged into it, when really it’s just using the shared network drive.
Vulnerable conditions
This is where we found our customers with the mysterious hacks – their websites would run on a server with a network file system. Each of the three hosting platforms used NFS, and had a variation on the same problem. Three conditions existed which combined to make the platforms vulnerable:
- Customer files were stored on a shared file system (in this case, NFS).
- All directories in the shared file system were world-traversable, and customer files were world-readable by default.
- The full path to a customer’s website directory was public or could be guessed.
The vulnerable hosting platforms all stored their customers’ files on a shared file system. The topmost directories were owned by the server root account and not set to be world-readable, so regular user accounts were not able to list their contents. However, for various reasons, all directories were world-traversable.
When hackers compromise a website, it’s very common for them to try to explore the server and see if they can locate other websites where they can create files, or at least read the contents of sensitive files like wp-config.php. The vulnerable servers did not allow for listing of the shared file systems, but their customer directories were set up so that the names of customer directories could be guessed. At some point, the hackers realized this, so they were able to use one hacked site to read sensitive files out of many other sites.
Hostway
On Hostway, the site files were located in a directory structure like this:
/home/14/23/1012314/web
The /home folder contained dozens of directories, and each of those contained multiple directories like 1012314, all following the numerical naming convention. The folder 1012314 and everything in it belong to the user; the directories above it belong to the server root account.
If a hacker took over the website in 1012314, and tried to explore the server by listing /home, they would get a “permission denied” error. They would also be denied if they tried to list /home/14 or /home/14/23. However, they must have realized that all the customer directory names followed a certain pattern:
/home/XX/YY/ZZZYYXX/web
XX and YY are two-digit numbers, and ZZZ is a three-digit number. So the hacker, having compromised just one site, could run a script from it to try all possible numbers and try to list files in each path. For example, it might start with /home/00/00/0000000, then /home/00/00/0010000, etc. Any incorrect guesses would result in a “no such file or directory” error from the server. But when the script correctly guessed an existing directory belonging to some other customer, the server would duly list its contents – because all customer directories were world-readable by default, and all directories above them were world-traversable.
So the hacker could find all the directories in the shared file system, each of which housed at least one website. If the website happened to be WordPress, he could read wp-config.php and get its database username and password. And since he already has control of a legitimate website, he can use it to reach the database server.
Using this method, the hacker could tamper with the databases of virtually any WordPress site running on the same shared hosting platform. This would all happen through the original hacked site, and nothing would ever appear in the logs of the other sites.
Momentous
The Momentous server platform (Rebel et al.) had a similar problem. The names of directories in the network share were random numbers, like /nfsvol-c/54/54321, and they weren’t world-readable but were world-traversable.
But when a website was created, both the site’s directory and the user account associated with it were named after the site. For example, if you set up example.com, then you might end up in the directory /nfsvol-c/12/12345/example.com/public.
A hacker trying to list the contents of /nsfvol-c/12/12345 would be denied. But he would be able to list the contents of /nsfvol-c/12 and see that it contained a directory named 12345 belong to the user account example.com. So he could correctly guess that there was a directory located at /nsfvol-c/12/12345/example.com/public and read the files within it.
Paragon
Likewise, the Paragon platform (Vidahost, TSOHost, et al.) had site directories sorted alphabetically. If you set up example.com on their platform, its files would be in the directory /var/sites/e/example.com/public_html.
Finding other sites in this case was trickier, but still possible. An attacker could take over a site on a Paragon server, and use DNS tools to find other WordPress sites running on the same server IP address. For example, if they found that otherexample.com was running on that server, all they had to do was look in /var/sites/o/otherexample.com/public_html for the files.
Remediation
One of the worst things about this vulnerability was that there was little that a site owner could do to prevent exploitation. In theory, removing world-read permissions from wp-config.php would protect a site, but there was no way to guarantee that some script or process wouldn’t reset it to the default vulnerable permissions. Moreover, only a small minority of site owners would know the adjustment was necessary. The only reliable defense against the exploit was for the hosting companies to fix the vulnerability.
We first contacted Hostway about this in September. At the time we did not have a formal way of dealing with service vulnerabilities. In December we updated our vulnerability disclosure policy to include them. On December 15 we formally contacted Hostway under the new policy, giving them a 14 day disclosure deadline, as the vulnerability was being actively exploited. They deployed a fix on December 21.
We also notified Paragon on December 15. They replied that they were developing a patch already, and requested more time before we published details of the exploit. On January 29th, Paragon confirmed that they had deployed a patch the previous month and had taken additional action to help remediation. We subsequently verified our proof-of-concept no longer worked on their servers.
Momentous received the disclosure on January 2, and deployed a patch on January 15.
What you need to do
If you use shared hosting on any of the companies we mentioned, use Wordfence to check your site for issues. We’re still getting a few site cleaning cases each week where the customer’s site was exploited before the fix was deployed, and the malicious code has remained on their site since then. The most popular malware we saw was obfuscated Javascript code inserted into every post, like this:
Sometimes cryptomining code was loaded into the page, other times it simply redirected the user to a scareware or spam site as soon as they clicked on anything.
We also saw a lot of sites with cloaked pharma spam links scattered throughout the posts. Their format varied, making them tricky to remove.
If your site has these issues, we recommend using our site cleaning service to fix them.
Conclusion
We are pleased with the positive impact adding service vulnerabilities to our Vulnerability Disclosure Policy is already having. The hosting companies we have worked with have been generally responsive, deploying fixes to issues that were causing many WordPress sites to be hacked.
With the popularity of WordPress today, the security of the WordPress community at large is critically important. We are pleased to see that our new approach is working to support that need and bringing about an improved overall security posture for the community.
Our Security Services Team continues to analyze hundreds of hacked websites each month, so we expect to find more of these on an ongoing basis. We will continue to provide updates here on the blog.
CORRECTION: An earlier version of this post mistakenly said that Paragon had not notified us about deploying a fix. This has been corrected to state that on January 29th, Paragon confirmed they had deployed a patch the previous month and had taken additional action to help remediation.
Note: All product names, logos, and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.
Comments
1:24 pm
I'm both disgusted and relieved to read this. I identified this issue with TSO Host back on 22nd September 2017 and was put through weeks of hell until I figured out the issue and solution.
Wordfence previously mentioned in a blog post before Christmas of issues with hosts but couldn't identify the hosts for understandable reasons.
TSO Host put me through hell, claiming nothing was wrong, they were not compromised and that the fault lay with me or WordPress. Here is the truth in black and white!!!
Thank you WordFence for doing what you do, I can't wait to share this post with TSO.
2:47 pm
Hi Simon. We added a section to the post titled "A Note on Disclosure And Responsible Vendors" after your comment was posted. I just wanted to note that here so that other readers have that context. Sorry to hear about your experience. We are happy that TSO was responsive when contacted by us and appear to have resolved the issue.
3:28 pm
Hi Mark,
Just read the addition and thanks for allowing my original comment. It's a shame TSO didnt listen to me originally but only listened when contacted by WordFence in December months after I notified them of the issue and provided them with evidence including support from Sucuri.
However, like you say I'm glad they finally listened and patched things. It's great to know that Wordfence has added this type of monitoring and analysis now for hosts and that responsible disclosure is used. I wouldn't expect anything else from a team of such high standards.
Please don't consider this shaming at all, I just felt a need for more information and transparency as this particular host was mentioned and they didn't even notify you that their network was patched if I've understood your article correctly.
Keep up the amazing work!
3:55 pm
Thanks Simon. We very much appreciate your comment. It's good data for us.
3:49 pm
My client got hacked several times on Hostway and neither Hostway or Wordfence's cleaning team figured out this was the issue at the time. We changed hosting providers and have been safe ever since. I'm glad you found this both to validate my diagnosis that the weak link was Hostway and so that Hostway's customers will now be safer.
4:04 pm
I had a site hacked on Saturday 3rd feb, every post_content entry had 2 scripts serving ads added. Scans of the site found nothing.
This was on a Vidahost server. Makes me wonder if the patch was deployed by then, if it was effective.
4:25 pm
Hi Mike,
This is exactly how the sites were being defaced and injected with spam on TSO Host, same family as Vidahost ( Paragon ).
Vidahost have maybe been slower to respond. Set your wp-config file to permission 400, it will stop future compromised for this specific issue.
You can also run an SQL search and delete command in your database to remove the injected code from the post_content table.
Kind regards,
Simon
4:12 pm
Well, it is nice to hear that some web hosting companies are responsible and fixing bugs when told to do so. But could you share some details?
What was so specific about NFS in this case? I think the same problem could happen on local fs as well.
What was the fix? Remove world read/traverse permissions on document root?
Other responsible web hosting companies might benefit from this information and preemptively fix this even if the bug is not abused on their servers yet.
4:14 pm
I've posted your question on our internal Slack and will post any follow-up.
4:47 pm
Hi Vladislav,
I got a reply from Brad even though he's in his car right now. The one thing that was common across hosts is that they were all running NFS. Honestly if I had to repost this I would remove NFS from the title. The reason I think Brad and Dan (the authors) added that is because all three hosts were running NFS, it added complexity in fixing the issue and Brad tells me that at least one host had to apply a kernel patch to fix the problem.
And to be clear, it is a permissions problem. Brad told me in my chat with him a few minutes ago he suggested fixing it using groups but for some reason they had to make other changes.
We don't have direct access to these hosting provider's infrastructure. We only know what they tell us and we can only access user accounts to evaluate security. So what we saw was repeated hacks, and then we evaluated security from a customer perspective and discovered the underlying cause and reported it.
The fix is to lock down FS permissions so that you don't have cross-contamination between accounts. The post gives you a reasonably good idea of what the permission holes were.
Mark.
4:48 pm
I'll guess that all that was needed was: open_basedir /docroot in the apache virtualhost.
And of course propper file permissions.
7:45 am
I also think that "openbasedir" would solve this problem.
4:14 pm
I feel your pain Simon. It's very frustrating trying to work with someone to discover a problem of any kind in the spirit of co-operation, and to be told their side is problem free. Leaving "us" to find a solution we either can't fix or can't see. The only after stressing to discover that the problem was in their side the whole time. I'm not sure that makes for responsible HOSTing. However, I do accept it was hard to find this vulnerability, and it probably seemed the problem was not in the HOSTs control at the time. I guess my wish was that providers are not so quick to brush off reports of issues that they should be looking into, sooner, and maybe report looking rather than not looking for a problem.
In any case thanks WORDFENCE people for your approach to the problem.
It's always best when we can work together in a non destructive manor to fix vulnerabilities.
4:26 pm
Was this an actual vulnerability in NFS or was it a permissions/configuration issue? I would hesitate to call the latter a vulnerability. Thanks; this is a great report and your team is doing some great work.
4:41 pm
Hi Tom. We didn't say it was a vulnerability in NFS. All three hosts were using NFS for their filesystems which added complexity to their configuration that made it harder to configure and ultimately fix the issue. At least one host told us they had to apply a kernel patch to fix the issue - and again, I don't have the details unfortunately. I agree that this is a permissions/config issue.
I'd also add that we have limited visibility on the hosts configurations - we just have the ability to audit security on their systems and then all we know about their config is what they tell us.
6:12 pm
You should examine Bluehost, when I had web sites there many were hacked, and I never got an explanation. While they may have patched the issue, I bet they have not, considering how many horror stories I read on the net.
Thanks for a great product!
7:32 pm
Thanks. I'm not currently aware of any outstanding issues with Bluehost. We'll be working with any hosting provider where we notice a trend developing.
Mark.
10:25 pm
Hi,
I have about 24 websites I host with Tsohost and was always singing their praises until recently. Since the end of last year, two of my clients sites had been hacked and injected with the ad spamming js. After many, many hours chatting to Tsohost about this, their only response was basically “it’s not our fault/problem, you should get a developer to help you”. I’ve basically spent the last 6 weeks, firstly on one site, then another trying to remove the script myself. Restoring the sites from back ups, editing PW’s etc only to find they get hacked again within days. I had to shell out almost £200 to get the first site cleaned by sucuri but can not afford to do that wihth them all PLUS have spend hours upon hours trying to remove this myself. I even totally rebuilt one of the sites from scratch and within a few hours it was hacked again. To say I am furious is putting this mildly. Tsohost insisted there was no vulnerability on their side and laid the blame solely with me, and really offered little help with trying to find a solution. I am still trying to remove the script from the 2nd site myself as I cannot afford another £200 to pay sucuri but don’t know exactly how to remove it from the DB. Any advice would be greatly appreciated.
I will contact Tsohost with this knowledge to let them know how disappointed I am with their narrow mindedness to this situation when all along the vulnerability was on their end and they were aware of it. Regards Cammy
12:00 am
Sorry to hear that Cammy!!
12:43 am
Hi Cammy,
Set your wp-config file to permission 400, it will stop future compromised for this specific issue.
You can also run an SQL search and delete command in your database to remove the injected code from the post_content table.
Kind regards,
Simon
10:42 pm
Mark, is it possible for a host to request one of these audits? I'm in the midst of setting just this type of thing up and want to be sure I get this permissions issue correct before putting customers on the servers.
11:58 pm
Hi Scott,
I'll ask Brad and he'll get back to you.
Regards,
Mark.
10:49 pm
I don't get how this is related to NFS. The same permissions problem can be there even if no NFS shares are in use.
11:57 pm
Agreed. If we had a do-over I think we'd probably re-title this post. As I've mentioned to previous commenters, NFS was a common element between all three hosts. It also added complexity etc. but this issue could exist without it. At it's core, this is a file permissions issue.
12:27 am
I used to host my website on TSO host server until it repeatedly starting getting hacked and injected with malware. After countless emails exchanges with their support they couldnt fix the issue. Then i hired a security expert and he suggested me to change the hosting provider immediately. I even sent his findings to TSO host but they refused to believe that they have got security issues. They used to have amazing support. Now all they have got is unsupervised chat bot on the website where you can leave the message but do not expect any reply.
12:33 am
Thanks for the nice work!
Some hosting companies actually still have instructions & help files instructing customers to configure with 777 permissions ...
I remember having issues with MediaTemple years ago that sounded very similar to this vulnerability, but they were amandant that they were not at fault ... however the problems went away when I changed the hosting company. I just got tired arguing with their support that was effectively blaming me.
A closely related problem is the plethora of bad advice on help forums where users are told to manually set 777 permissions to solve server issues.
And a lot of software (like Sendy : chmod 777 /var/www/html/uploads) instruct the user to set 777 permissions to some directories, which I don't find a very good idea at all.
Personally, I stay away from software and hosts that want 777 permissions anywhere, because it destroys my confidence in their security awareness.
3:18 am
In my personal opinion for a decent sized hosting group Paragon are a very dangerous company to trust. Their email platform is extremely unsafe too and full of spam (probably from all the hacked sites?) We switched away from them a good while back and barely had an issue since. Like Simon mentioned above, they are very quick to dismiss any issues and blame the user rather than launch an investigation into the reported issue. Would also like to note that our WordPress sites we used to host with them also had the same issues as reported in the post and never had a solid explanation as to why. Glad to finally know what it was! Great work as usual Wordfence!
As a final note, you guys should consider a system for the public to notify you guys of any issues like this in the future, hosting companies, developers etc clearly listen to WordFence much more than joe blogger.
3:58 am
What about "client shaming" and "client blaming"? Our reseller account on Vidahost was hacked in July 2017. We reported all these details (erroneous user accounts and remote MySQL IPs being added and restored even when deleted; changes to the database including changing from UTF-8 to UTF-7) to the hosting company and paid Wordfence to check out it out. The Wordfence team investigating advised it was server-side. Vidahost's response was to blame us and double our hosting fees because we'd asked for so much support in resolving the issue! The disruption for clients was massive. Resellers like us became 'piggy in the middle' between an unsupportive hosting company and distressed clients. Wasting hundreds of hours and a lot of money trying to fix something that was not our fault. We did all the right things. Sites always up to date and regularly scanned - manually checking and removing erroneous database entries, RemoteMySQL IPs and user accounts; replacing all the files, changing all the passwords and salts on all the sites on the reseller account, paying for Wordfence to check the site - only for sites to be hacked almost as soon as they had been repaired/restored. Maybe it is time hosting companies learned to listen to their clients and stop blaming them when faced with evidence, or clear indications, that indicate a server-side attack.
6:33 am
I make your blog required reading. There are not many that are worth my time, but you are always having something good. Thanks for being vigilant and watching out for us devs in the trenches! Now, I'm going to examine my host : )
6:46 am
Great post Brad. I ran into this issue several months ago with a client hosted at Birch/cBeyond. Your post validates my suspicion (at the time) that the database hack came internally via the shared server. We were able to remediate the hack by removing the hacked-in admin users, changing the database credentials and removing the defacement messages.
8:24 am
Update to my earlier comment -- woke up today to a Wordfence email notification that an admin user was externally created on this website. I contacted Birch/cBeyond to report my concerns from the Dec hack and again advise that this is likely coming internally via the shared server. Birch/cBeyond hosting is provided by Hostway -- which the article says was remediate.
Brad/Mark -- Perhaps there's still a hole?
5:39 am
I'm hosted with VidaHost - part of Paragon.
I've had two sites that I had in development on subdomains under the live site and used the Duplicator plugin to move them over. Duplicator had an XSS vulnerability in it late last year.
The subdomains did not appear to be infected until they were made live.
I wiped both sites, but kept the hosting the same and when I reinstalled both sites from a clean Wordpress upload, they both became reinfected within hours even though every single password had been changed.
Just a few days ago - in Februrary - the Chat representative would not understand that from my experience it actually meant that the invisible files that I couldn't see must be infected.
The only solution, because VidaHost couldn't help apparently, was to actually scrap both domains completely and start again fresh.
A monumental waste of time!
5:04 pm
We are proud to present to our customers the wordfence application that has always met our expectations and surprises every day in protection by helping our internal tools. Congratulations to the solution developers and staff.
6:24 am
Hi everyone, as an update to this regarding TSO Host, I feel this is important and not shaming, especially as if I have understood the blog post correctly, TSO did not notify Wordfence of any updates to their network.
I submitted this blog post to TSO and the official response I got from them is lengthy, the key thing is this part of their response:
"The article from Wordfence is something that on our system was as it was designed. I have been told by our senior admins and developers that this is something they had recently looked into but do not feel it to be as much of a risk as Wordfence have made out. We have though made some changes to reduce risk here."
So not only do TSO think that the issues WordFence identified are not serious. TSO still stated also in their response to me they are not taking responsibility and it's not their fault.
1:26 am
Just wondering if your cleaning service will clean files outside of Wordpress. We have a WP site that has been hacked and it appears that cPanel has been accessed. There are multiple directories and files created outside of WP. Will your cleaning service be able to help with this?
5:11 pm
Hi Karen,
A reply from our team: "Yes, we’ll clean all of the publicly accessible files. If those include another CMS (additional WordPress install or Joomla install, or something else), there is an additional charge. However, we have volume pricing for such things."
Regards,
Mark.
1:52 am
We had a customer site (hosted at TSOhost) compromised on Monday 19th Feb (so after the announced Paragon fix). However, my guess is that the attacker had got the DB credentials from the wp-config.php before the fix and only used them on Monday. So, I would suggest you advise that all DB passwords be changed on any site using any of the hosting companies identified - otherwise an attach could still happen.