Episode 94: Hosting Provider Exposed 63 Million Customer Records

A hosting provider exposed over 63 million customer records via an open elastic search database containing verbose logs with plain-text username/password credentials for numerous WordPress, Magento and other sites. We also talk about the security updates in WordPress 5.5.2/5.5.3 and the accidental 5.5.3-alpha autoupdate.

We talk about object injection vulnerabilities like the one discovered in the Welcart e-Commerce plugin and how POP chain attacks work.

And Google’s Project Zero finds a high-severity vulnerability in GitHub Actions not fixed within the 90-day disclosure grace period.

Here are timestamps and links in case you’d like to jump around, and a transcript is below.
0:00 Wordfence is hiring pentesters, a security/operations engineer, and security analysts
1:11 Hosting provider exposes over 63 million customer records
6:44 WordPress 5.5.3 emergency release
11:47 The security fixes in WordPress 5.5.2
15:45 Object injection vulnerability discovered in Welcart e-Commerce
18:21 Google Project Zero discloses vulnerability found in GitHub Actions

Find us on your favorite app or platform including iTunes, Google Podcasts, Spotify, YouTube, SoundCloud and Overcast.

Click here to download an MP3 version of this podcast. Subscribe to our RSS feed.

Episode 94 Transcript

Ram Gall:
Hello, and welcome to episode 94 of Think Like a Hacker. The podcast where I, QA Engineer and Threat Analyst Ramuel Gall and Director of Marketing Kathy Zant, talk about tech and security. And I hear we’re also hiring, in case you want to work with us.

Kathy Zant:
We are hiring, we have a number of jobs open. I think we are looking for folks who can help clean up some hacked sites and we’re hiring pentesters to do some penetration testing, which sounds like loads of fun and a security operations engineer to help us with securing our network and all that fun stuff. So all of that’s available, if you just go to Wordfence.com and look for the “careers” link in the footer, you can read about those jobs and apply. So we would love to hear from you.

Ram:
We really would, because this is an awesome place to work.

Kathy:
It is, and we have some great benefits. We’re all remote. And we also have good jokes, right?

Ram:
We do have, we have the best jokes. We have dad jokes. We have more dad jokes, every week, twice a week, Tim comes up with a dad joke.

Kathy:
Exactly we have a lot of fun, security is important, serious business, but we like to have fun while we are helping keep WordPress secure. What’s our first story? It looks like we have a hosting provider that exposed 63 million customer records. What do you know about this, Ram?

Ram:
So the good news is that’s not necessarily the records of 63 million customers, but it is 63 million records. So I guess security expert Jeremiah Fowler posted on his blog that he found an open Elastic search database. Elastic search is a large scale database you can use to search through lots of data really quickly. This is not unlike those open S3 buckets of yore that led to so many other data breaches back in the day.

Kathy:
Right.

Ram:
But I guess he found it on October 5th and it was a hosting company called Cloud Clusters Inc. Though, I guess they had a number of other companies with similar names that are likely owned by the same person, but apparently they’ve been in business since 2017. They don’t really have a lot of social media followers.

Kathy:
Yeah. I took a look at them and all I could find was a Facebook page and they had, I think, eight followers and they have only started posting this year. I went on Archive.org to see how extensive they’ve been around. And I saw one snapshot in late 2019 and a few in 2020. So they seem like they’ve only been in … at least marketing their business this year. But according to their website, they have data locations in Oregon, North Carolina, Colorado, and Texas. And what did we see with the records in this database?

Ram:
So this was where it got kind of sketchy. It looks like they had clear-text credentials for various CMS systems like Magento, WordPress, or even just for my SQL databases of customers hosted with them. Which is bad because even in the WordPress database itself, passwords are still hashed. So I guess they had services running using the like clear-text passwords, and then they logged all the commands they ran that included those passwords. And those logs were at least part of what were exposed.

Kathy:
Oh, no.

Ram:
If you or anyone you know, have had dealings with the Cloud Clusters Inc. or any of the other potential company names like MGT Clusters, Hyper-V Mart. They had a couple of different variant company names apparently. But yeah, if you’ve been hosted with one of these guys, you may want to be aware of this. Especially if you’re in California, it looks like they still haven’t necessarily notified their customers and this was on October 5th. And if they have any customers in California or Europe, they would be subject to laws mandating that they reported to those customers. Though, they really need to report it to everyone.

Kathy:
Right. So if you are a customer of this particular hosting company, there is no way for us or anyone to really know if malicious actors had accessed this database, made a copy of it, downloaded it somewhere. Maybe it’s on the dark web somewhere. We really don’t have any information about that. So we have to assume that these have been exposed.

Ram:
He did say that he found evidence of a Meowbot attack. That’s basically just a script that deletes data. So it does seem like threat actors, may have been interacting with this information. Though, it’s hard to tell what capacity that was in. Verbose logging can be good, but be super aware of where you keep your logs, is kind of the takeaway from this. Since a lot of this was basically log entries that had problems. If anyone can access your logs and if you’re putting sensitive information, like running commands on the command line that include the password in the command line, which most command line utilities tell you not to anyways. I mean, it does seem like a lot of things had to go wrong here for this to happen. It wasn’t just one thing.

Kathy:
Right, right. So there’s a number of takeaways from this. First of all, if a security researcher contacts you and says, “Look, you’re leaking information out into the web.” There are some steps you have to take as the owner and the person or the organization that is entrusted with that information of your customers. You basically owe it to your customers to let them know. Right?

Ram:
Exactly. And I mean, that’s even if you don’t have any customers in California or Europe. Though, I think most internet companies at this point will have customers in at least one of those places.

Kathy:
Exactly. So you’re required by law to be transparent, but it’s also just good business to establish that kind of trust. If you are a customer and you’re exposed, this is not necessarily something … If you’re hosting provider or anyone contacts you saying, “Hey, look. You’ve been involved in … Your information has been exposed in a leak or a breach or something like that.” That’s a good sign, right?

Ram:
Yeah. It means that this company takes your security seriously to enough of an extent to let you know that there was a problem. Because if you know there was a problem, you can update any credentials. And that is really in almost all data breaches, except for a few like Equifax. The username-password combinations are the things that are really a value. Especially if they can be used to actively exploit existing sites and gain access to those.

Kathy:
Right? And if you are running a WordPress site with WooCommerce and your customers’ data is being stored in that database, or if you’re running Magento and your customer’s data is being-

Ram:
Then all of the sudden it becomes your problem.

Kathy:
It does. It does, because you are entrusted with your customers personally identifiable information or PII. And that means that you have to then take steps to let your customers know that they have been exposed.

Ram:
It’s just the gift that keeps on giving.

Kathy:
That’s what security is these days, isn’t it?

Ram:
It really is. Speaking of, and I hear there was a bit of a hoopla with the WordPress 5.5.2 and 5.5.3 autoupdate and emergency release. I know I was on vacation for that whole thing.

Kathy:
So you just sat with popcorn and watched us all scrambling to research this.

Ram:
I went through like three bags of popcorn.

Kathy:
So now we’ll measure issues in the WordPress world by bags of popcorn that Ram can consume. So what happened with this 5.5.3-alpha release on Friday?

Ram:
So from what I understand, the 5.5.2 version, which we will totally dive into right after this, had a severe problem where if you try to install it on a fresh website and you hadn’t already set up database access and credentials, which I mean, most people don’t necessarily have set up. Unless it’s some sort of automated installation and even then not all of those do that, it just couldn’t be installed. So I guess what happened is between 3:30 and 4 UTC, that is Universal Coordinated Time, on October 30th, a bunch of sites got automatically updated from 5.5.2 to 5.5.3-alpha.

Ram:
I guess this happened because they disabled the 5.5.2 release because of that bug, which means that the next thing in the most recent version stream showed up as 5.5.3-alpha. And it’s like, “Hey, we should automatically update to this.” And very fortunately there were no issues. It came with a few extra bells and whistles, like all the 2020 themes and Akismet.

Kathy:
Everyone needs a new, fresh copy of Akismet. Right?

Ram:
Yeah, everyone apparently does.

Kathy:
Yeah. Well, so functionally 5.5.2 and 5.5.3-alpha-49449, essentially the same code, right?

Ram:
Yeah. It wasn’t actually a disaster and this is why we emphasize testing things a lot because no matter how much they test, and I know the WordPress Core team does a really good job of testing and a really good job of making sure that everything has processes in place to prevent issues. They can still happen, but more testing totally reduces the likelihood of this kind of thing happening, so it is important.

Kathy:
People reaching out to us on Friday asking what was going on and what our take on it was. Obviously we’ve talked about auto updating quite a bit, not only on this podcast, but we did a Wordfence Live recently on an auto updates that were added into WordPress 5.5. And so people are concerned, should I allow auto updates, if things like this are going to happen? What’s your take on that?

Ram:
I think recently they’ve done more good than harm, but this does showcase that even with processes in place, there is the potential for unexpected behaviors. I think with this specific thing it … I don’t necessarily want to say it was unforeseeable, but it certainly seemed to be unforeseen just because I don’t think anyone anticipated that particular course of events happening. So the good news is it probably won’t happen again. The bad news is, I don’t think anyone really had a choice in it unless they specifically hard-coded update disabling into their WP config to like not update core updates. Which actually puts you at greater risk, unless you’re meticulously paying attention to which updates are happening. Which if, you have a large-scale operation and a staging site and a team that can review code, that’s probably a very good idea.

Kathy:
Exactly. Yes. I mean, obviously security is a set of scales of weighing risks, right?

Ram:
Yeah, there’s no 100% secure. There’s no such thing as perfect security. It’s risk management. And our risk is not the same as your risk.

Kathy:
And a large scale or operation with a site that’s getting 10,000 visits per minute is going to have a different set of risks altogether. So you have to take all of these things into consideration, weigh out all of your scales of security and decide what’s right for your particular install. Obviously, WordPress being the most used CMS, we have tons of people who just have a website and a side gig, and that’s their side gig and they’re at work. And if a security release goes out and they’re not going to have time to manage that update, auto updating is the right solution for them. Making sure that auto updates are happening so that their site, their asset, doesn’t get hacked while they’re doing something else if they don’t have a security team, obviously that’s what goes into their scales.

Ram:
So, I mean, I have auto updates turned on, on my site because I don’t actually look at it nearly as often as I should. But we would definitely not want auto updates going on on like, a large company’s main site.

Kathy:
Exactly. Exactly. Well, I use Wordfence Central, so I manage all of my security.

Ram:
And it gives you alerts when there’s updates available.

Kathy:
It does, and I’ve got premium on there. Yeah, I’m set. I can handle my updating. Anyway.

Ram:
Yeah. Speaking of which, it seems like the security release, the 5.5.2, which kind of got lost in the-

Kathy:
Kerfuffle.

Ram:
That was the word I was looking for. I was thinking either that or hoopla. But a lot of the time when we see security releases in WordPress Core, there are things that take a really like specific set of circumstances to exploit. And they might not always be terribly realistic, but just because of the sheer install base, you can’t be sure that they’re not going to happen. I feel like this one actually had a couple of exploits, or patches, that were interesting because they were maybe a little bit more relevant and a little bit more exploitable than a lot of the recent ones have been.

Ram:
The first one, it wasn’t actually a vulnerability that they patched. But we’ve talked about object injection before and how you can inject a PHP object anytime you unserialize user input, but you can’t really do anything without it, unless you can build what’s called a POP chain. So object injections are usually a little … I mean, they can be super high severity, but they’re usually not as much of a worry because an attacker has to do a bunch of research on your site to make sure that they can make a POP chain that you have in plugins or themes or something installed that they can make a POP chain with. If you have something in WordPress core that has the potential to be a POP chain, all of a sudden, any object injection vulnerability just became a lot more serious because it means that attackers can possibly take advantage of that.

Kathy:
Right. And that could lead to remote code execution and takeover of a site. Right?

Ram:
Exactly. In this case, the actual proof of concept was remote code execution. There is some good news and that’s, that the proof of concept strings that they came up with at least … Now, I can’t say for sure that all potential exploits of this would be the same way, but the actual strings of proof of concept, they use failed a check that WordPress uses to tell if data is serialized or not. So a lot of plugins we’ve seen that have object injection vulnerabilities, use that check or use maybe_unserialize, which uses that check before they unserialized stuff. So this would have failed that check. So it was a little bit safer, a little bit less bad than it could have been, but still the worst case scenario was very bad indeed.

Kathy:
Got you. Okay. And there was another vulnerability here that was patched in 5.5.2, that you found interesting. This had something to do with $pagenow?

Ram:
Yeah, WordPress uses a global variable called $pagenow to figure out what page it’s on right now. And it also uses this to figure out the current screen global variable, which is another thing that a lot of plugins, a lot of themes use to figure out what they should be showing you at the moment. Turns out that you can inject user input into those things. Because of the way it’s used by a bunch of plugins and themes, there’s probably not a great way to completely filter out that user input without breaking compatibility. So what they ended up doing was escaping any places in WordPress core on the dashboard that use the output stuff based on those. So it couldn’t be used for reflected cross site scripting.

Kathy:
Okay. Good deal. So those are the two highlights of the 5.5.2?

Ram:
Those are the two I found most interesting. I mean, there were a number of other things that were potentially more … A few of them were potentially more exploitable than usual. A few of them were more or less run of the mill, only very specific circumstances, but those are the ones that stood out at me.

Kathy:
See for me, the only takeaway is, okay, this is a minor release. My sites, most of them auto updated. A couple I had to update myself, but just making sure that you are getting these updates on your site. Especially, so the disabling of deserialization, I see that more as a security enhancement, but with the object injection vulnerability that you just wrote about this morning with the Welcart eCommerce plugin that-

Ram:
Yeah, I actually have a blog post out this morning about an object injection vulnerability in an eCommerce plugin that is big in Japan.

Kathy:
It is big in Japan.

Ram:
At least that’s what they say, and I believe them.

Kathy:
Yeah. So I have a question for you and I didn’t ask you this before, so if I had a site and I had this vulnerable Welcart object injection vulnerability on my site and a previous version of WordPress that wasn’t patched, would somebody be able to take advantage of that and get remote code execution?

Ram:
So the good news is no, in this case. Because that Welcart plugin, it did actually use that check to make sure that the string was serialized. And because of the way WordPress implements that check, the string would have failed it, even though it was actually serialized and would unserialized into a PHP object, it just fails that particular check. So in this case, this particular POP chain, couldn’t be exploited. Though, some of the object injection vulnerabilities I’ve discovered in the past may have been able to take advantage of this. And I’ll have to revisit those and check, but I seem to recall at least one of them not using that check.

Kathy:
Okay. And explain again, what a POP chain is. We’ve talked about that a little bit, but if you can explain it for our listeners.

Ram:
Sure. Every time you unserialize user input in PHP, you can basically feed it a string that gets unserialized into what’s called a PHP object, which is a data structure, a live data structure. Now, if there’s any active classes, any code running that matches up with that data structure, and it has certain things called magic methods. The destruct magic method, the wakeup magic method, if it has any of those and they do anything interesting, like deleting files in order to clean stuff up or running one variable against another. You can change what’s in those variables, so when it does that thing, you have control over what it’s doing. So it says, “Instead of deleting the log files, I want to delete wp-config.” Instead of running a cleanup function, I want to execute a function that injects code into another file and executes it.

Kathy:
So now with this one, it looks like the firewall rule is going to become available to free users next week sometime. So everybody who was running this will be protected by our firewall soon, but they should still update.

Ram:
Always update.

Kathy:
Excellent. Okay. And our last story, Google, it looks like they found a vulnerability in GitHub. What do you know about this?

Ram:
So this was another Project Zero find, I feel like we bring up one of their finds almost every week just because they are the heroes of InfoSec. And they’re always finding cool stuff that is incredibly terrifying. But in this case, GitHub has something called GitHub Actions, which are used for continuous integration. But it basically lets developers automate the workflow. So if you’re committing code, you can write code that tells GitHub what to do with that code. Like let’s say, you’re writing in a compiled language. So if you’re committing code, you want to make sure that whatever you just committed can compile and make sure that it passes all of your automated tests.

Ram:
And you can use GitHub Actions to run all of those things on the code you committed. The main problem is that the Actions kind of look at the output of each set of code and look for instructions on what to do next. Which means that depending on how you set up the Actions, like if you set it to like do verbose logging of what you just did, people could submit issue tickets with instructions in the ticket titles. And if your actions printed out the ticket titles of the things that you were working on, then it could execute code based on those actions. And remote code execution is always bad, if someone you don’t want to have ends up getting it.

Ram:
So a public proof of concept hasn’t been entirely made available beyond just the basic gist, because apparently you can set environment variables by including them in issue ticket titles, as one of the things that was shown. But one of the worst things is that Project Zero disclosed this to GitHub 104 days before they actually made it public. And GitHub still hadn’t fixed it.

Kathy:
Yeah. Typically, Google Project Zero, it looks like they have a standard 90-day deadline and they state that over 95.8% of flaws are fixed within that deadline. But it looks like GitHub was taking a little bit longer.

Ram:
Yeah. You would think that they would have their stuff together, but apparently they didn’t. Apparently also their CDN also had an expired certificate around the same time. And yeah. It looks like that was just not a good week for GitHub.

Kathy:
It does not look like a good week. So if somebody is-

Ram:
Or a good look for that matter.

Kathy:
It’s not good look either, because this is where so many people are keeping their code, maintaining their code, using GitHub and sharing capabilities of coding on so many different applications. And the fact that GitHub didn’t really prioritize fixing this is very troublesome for security-minded developers like ourselves.

Ram:
Yeah. And I mean, if you are running a private repo, you are probably less at risk, but realistically I think the main takeaway is to treat anything that a user can change as user input. There have been vulnerabilities in the past where people using fail2ban, which generates rules to block IPs automatically on the server side. There have been exploits against that because it generates rules by processing the IPs it sees and anything that a user can change is user input.

Kathy:
Exactly. Exactly. Very important, if you are using GitHub Actions, double check to make sure that you are not trusting your user input, no matter where it comes from.

Ram:
Yes. Verbose logging is usually good, but this seems to be another case where verbose logging is leading to more issues.

Kathy:
Yeah, yeah. Interesting. Okay. Well, that’s all of our stories for this week. Ram, thanks for joining me again on Think Like a Hacker. We will be back again next week. We will have more stories, make sure that you watch Wordfence Live next week. We’ll talk a little bit more about what happened last week with the auto updates and have some ideas for you and how you manage your WordPress sites and autoupdating on that particular episode. You’ll be there, right Ram?

Ram:
Yes, I will.

Kathy:
Awesome.

Ram:
Every Tuesday, Well, except this last Tuesday.

Kathy:
We need a break every once in a while.

Ram:
Yes, yes we do.

Kathy:
Great. Okay. If you want to follow us … If you want an email every time we publish to Think Like a Hacker, you can do that. Just go to wordfence.com/podcast. There is a form there where you can put in your email, and you will get an email when we publish. You can also follow us or subscribe to us on any of your favorite podcasting apps from Spotify, to Apple Podcasts, Overcast, Google Play, whatever. I mean, there’s so many of them now. Wherever you are, we are there too, and if there’s a story you’d like us to do a deeper dive on, please let us know. You can write back to us at feedback@wordfence.com and we will hope to cover your story.

Ram:
And go read my article. Also, we’re hiring.

Kathy:
Exactly. Go read Ram’s article about that … What is it again?

Ram:
Welcart, they’re huge in Japan.

Kathy:
They are huge in Japan, Welcart. All right, that’s it for today. We’ll talk to you next week.

Ram:
Bye.

You can find Wordfence on Twitter, Facebook, Instagram. You can also find us on YouTube, where we have our weekly Wordfence Live on Tuesdays at noon Eastern, 9:00 AM Pacific.