Episode 96: Hosting Provider Failures and Incident Response Preparedness

Two hosting providers experienced outages this week. GoDaddy had a brief outage affecting numerous systems on Tuesday, November 17. Managed.com had an extensive outage due to ransomware that affected all systems. We discuss what types of incident response preparations site owners should consider when events beyond their control occur.

We also discuss a large-scale attack targeting themes using the Epsilon Framework, the new head of security at Twitter, and an Android chat app exposing private messages.

Here are timestamps and links in case you’d like to jump around, and a transcript is below.
0:26 Large-Scale Attacks Target Epsilon Framework Themes
3:04 Ransomware attack forces web hosting provider Managed.com to take servers offline
6:51 GoDaddy had an outage
11:21 Twitter Hires Mudge as head of security
14:45 Android chat app exposes private messages

Find us on your favorite app or platform including iTunes, Google Podcasts, Spotify, YouTube, SoundCloud and Overcast.

Click here to download an MP3 version of this podcast. Subscribe to our RSS feed.

Episode 96 Transcript

Ram Gall:
Hello and welcome to episode 96 of Think Like a Hacker, the Wordfence podcast where we tell you about security, hacking, and other stuff related to security and hacking. I’m QA Engineer and Threat Analyst Ramuel Gall, and my co-host here is Kathy Zant.

Kathy Zant:
It is Kathy Zant, the Director of Marketing here at Wordfence. We’re really happy to be here. We have a number of stories about security and WordPress this week. Ram, you noticed this first story about this large scale attack targeting Epsilon Framework Themes. Tell us more.

Ram:
Maybe a couple of months ago, it was disclosed that I want to say maybe 15 themes all using the Epsilon Framework were vulnerable to what we call a function injection vulnerability.

Kathy:
What’s a function injection vulnerability?

Ram:
We’ve talked about object injection vulnerabilities in the past. Function injection vulnerabilities are kind of like a more powerful version of that. Basically with this an attacker could execute any public static function in a loaded class and provide parameters to it. So it’s a little bit more limited than a full remote code execution (RCE) since they can’t execute completely arbitrary code, but it wouldn’t rely on magic methods.

Ram:
For instance, if they were aware of a static function already loaded in most WordPress installations that could provide full RCE, they could take advantage of that and gain full RCE.

Kathy:
Okay. Were these attackers actually getting RCE on any of these sites?

Ram:
So there were about 1.6 million sites attacked on Tuesday, that was the 17th, in a single day. We had about 7.8 million attacks overall. But for the time being, it looks like they were just probing attacks. They were basically sending out requests against the vulnerable AJAX actions to see if a vulnerable theme was installed. Because if a vulnerable theme was installed, it would reply with a specific response saying that, hey, you need to provide the function you want to execute, rather than just a 400 bad request.

Ram:
We actually tracked it, and we’ve had attacks against 2.3 million sites at this point. That’s a majority of our user base. And I mean, clearly these attackers are not only attacking sites with Wordfence installed. The actual number of sites attacked is probably significantly larger than that because not everyone has Wordfence yet.

Kathy:
Okay. So if they’re just doing probing attacks, they’re basically collecting data maybe for another attack in the future?

Ram:
Yeah, that does sound like the case. And here’s the good news, Wordfence, even the free version of Wordfence, does protect you against these attacks. However, we still strongly recommend heading over to the article, checking to see if you have any of the themes mentioned there installed on your site, and updating it as soon as possible.

Kathy:
Good plan. And we will have a link to that blog post with the list of all of those themes in our show notes. So head to wordfence.com/podcast and find episode 96, which is this episode, and we’ll have links right there for you. Next, we saw a hosting provider, Managed.com, that has basically been down it looks like all week, hasn’t it?

Ram:
Yeah, I think so. Let’s check if it’s still down. Kind of looks like they’re still down.

Kathy:
What’s going on with this?

Ram:
Well, apparently it was ransomware by the REvil, REvil. Is that REvil or REvil operation?

Kathy:
I’d say REvil.

Ram:
REvil! In case you don’t know, ransomware is where an attacker gets into your system and encrypts all your data so that you can’t use it or access it and then demands a ransom in some kind of cryptocurrency, usually Bitcoin or Monero. REvil is a ransomware as a service. I guess they got their start back in April of 2019, and they’re currently one of the largest ransomware operations. They claim to have earned over a hundred million dollars a year in extortion payments, which means that some people are definitely paying their ransom. It looks like about a quarter of victims do.

Kathy:
Yeah. I saw an article on ZDNet when I was researching ransomware why it’s still so successful and how a hundred million dollars a year in extortion payments could have been collected by this one ransomware service. It looks like about a quarter of victims opt to pay the ransom in order to get their data unencrypted. Now, people are using Managed.com not only for website hosting, but they’re using it for DNS service. They’re using it for their email. And this attack started on it looks like Monday, November 16th.

Kathy:
So people have basically been down all week and don’t know what’s going to happen or how this ransomware got in there, but it’s definitely something that we’re watching.

Ram:
Yeah. It does look like the attackers are actually asking for under the market rates. They’re only asking for $500,000 when I guess the average payout is a million dollars.

Kathy:
Wow. What a bargain.

Ram:
Ah. Yeah. So how do you defend against ransomware? We keep on bringing up backups and specifically offsite backups.

Kathy:
Yes. You don’t want to back up to the same system that could be encrypted by ransomware. Definitely you want to have backups somewhere else. But I mean, these attackers are getting in with ransomware somehow. This means that they’re either exploiting vulnerabilities, socially engineering someone to exploit the human element of security. Either way, what would be your recommendation to protect against ransomware like this, Ram?

Ram:
Realistically, it could also be breached credentials, by the way. Those are extremely common. But on a just basic small host, small business level, it’s not likely to happen to your WordPress site. I mean, it can. I’ve seen it happen to WordPress sites, but it’s not that likely. But on an enterprise level, make sure that you have your users trained against social engineering. Make sure you require two-factor authentication for any accounts that have any real level of access.

Ram:
Make sure you have someone in charge of security at your company who understands these things and knows how to set policies to help prevent them. And make sure that person also believes in backups.

Kathy:
Backups, backups, backups.

Ram:
Yes.

Kathy:
So just basic security. You cannot operate any business in the world right now without having some basic security protections in place. And that goes for your systems and making sure everything’s updated, as well as your people and making sure that their systems in their heads are updated so that they’re thinking with a security mindset.

Ram:
Also, one of the things about ransomware is that it typically doesn’t happen all at once. Typically, it takes some time for them to get through enough of the network to make a difference. Which is why it’s a good idea to have a good incident response plan. And speaking of which, I hear GoDaddy had an outage and that was-

Kathy:
Ah, they did.

Ram:
…on the 17th. So yeah, that was-

Kathy:
Tuesday.

Ram:
Yeah, Tuesday.

Kathy:
It looked like it happened at the evening, about 7:00 PM Pacific Time. Even their homepage was down. So it looks like it affected a number of different systems. It affected hosting customers, as well as GoDaddy’s forward-facing systems. What do you know about this, Ram?

Ram:
So disclosure, I used to actually work at GoDaddy. They have a really good incident response team. I mean, that’s not to say that things weren’t pretty much always on fire, but that’s why they have a really good incident response team.

Kathy:
They’re the largest hosting provider I think in the world right now, aren’t they? In terms of number of sites that they host. I mean, they…

Ram:
It’s got to be either of them or AWS, but I think they’re probably the largest shared hosting provider in the world.

Kathy:
Yeah, definitely. They are a behemoth. So there are a lot of websites, WordPress and otherwise, that are hosted on GoDaddy systems. So with GoDaddy being down, a lot of customers were affected. Now, I’ve lived through a lot of internet companies who have gone through growing pains. I’ve had sites down for days at a time, and it’s incredibly frustrating for an end user. I mean, what would we recommend to our customers who are like hosted somewhere, that you have a site that’s receiving a lot of traffic and your customers are wondering what’s going on?

Kathy:
And this is beyond your control because your hosting provider is down and there’s not much you can do. What can someone do in that kind of situation?

Ram:
Having some degree of redundancy is a good idea. We’ll talk about things like having a hot site or a warm site backup. And although these usually refer to being able to move offices, you can sort of use that as an analogy. If you have backups in place, if you’ve got those backups collected somewhere safe and you think the outage is going to last awhile, and you still have access to your domain’s DNS, you can temporarily basically restore everything to a separate host.

Ram:
If your site is critical enough, if your site is mission critical, having that kind of redundancy in place is a really good idea.

Kathy:
Sure. Also, just having a place where you can communicate with your customers that isn’t dependent upon that site, isn’t depending upon that hosting provider, that you have social media in a number of places, that you have maybe a status dot your domain.com subdomain, as long as your DNS is not down, right?

Ram:
There’s sort of an in-joke in security and I think operations circles as well, and that’s that anything that goes wrong, it’s always DNS.

Kathy:
It’s always DNS. Yeah.

Ram:
It is always DNS.

Kathy:
Right.

Ram:
I mean, I’m kind of surprised that Managed.com wasn’t DNS. That time was actually attackers. That time was actually ransomware and not DNS.

Kathy:
Yeah. Yeah. But I mean, if somebody has an incident response plan in place for their business for an incident like this, something goes down that you have no control over, an intrusion occurs, you think through these types of events ahead of time so that you have written down somewhere, “this is what you do” so that you don’t have to like have clear thinking when everybody’s running around with their hair on fire, right?

Ram:
You follow the plan. You know who is supposed to be in charge of executing the plan. You know who to contact. You have phone numbers where people can be reached who are responsible for doing different parts of the plan. You have a way to set up a bridge call so that everyone can communicate and talk over what they’re doing to make the incident response plan happen.

Kathy:
Right. And also have a piece of that incident response plan be “how do you communicate to customers?” How do you communicate to the media if it’s a high profile type of attack.

Ram:
When do you communicate to customers.

Kathy:
Yes.

Ram:
How long does the outage have to go on before you’re like, we should tell people.

Kathy:
Exactly. Exactly. There’s a bunch of guides I found online that talk about incident response and best practices in developing incident response. So maybe we’ll throw some of that in the show notes as well.

Ram:
Yes. Are you going to be able to wake up your developers at 2:00 in the morning to fix stuff. And I mean, do you have an on-call rotation? That kind of thing.

Kathy:
Yes. All important things to consider. Hey, I bet Twitter’s got something like that now. What do you think?

Ram:
I’m sure Twitter has had something like that for a while, but it looks like their security is about to get a lot better. Famed hacker Mudge, Peiter Zatko… Here’s the thing, the names always sounded kind of familiar, but I actually had to look him up because he doesn’t really have a cult of personality going around and he’s just, by all accounts, just a super standup guy. I keep on looking at what he’s done. It’s like, oh, hey, I totally read his like buffer overflow exploit intro thing back in the day. And oh, he’s the guy who wrote L0phtcrack.

Ram:
And it’s just like, wow, this guy has been behind a lot of the like really interesting security innovations of the last few decades.

Kathy:
And security education. Twitter has hired Mudge as head of security, which I think is incredibly newsworthy for the security world. And I don’t think a lot of people… I mean, I was the same way as like, L0phtcrack? I remember using that way back in the day. It’s been around forever. I mean, he is a legend in the security world. But this is going to have a huge impact not just on the security at Twitter, which I think needs a little help. They’ve had a high profile intrusion that seems to have been from social engineering earlier this year.

Kathy:
I think it was this summer, July maybe. And now that he’s going to be head of security there, what are some of the things that you think that we can look forward to with Twitter and security there?

Ram:
Improving policy is honestly one of the biggest ways that you can make a difference. Improving user education. But he did propose confusing bad actors by manipulating the data they receive from Twitter about how people interact with their posts. If you’ve got a bunch of bots, maybe you might, for some reason, not want to suspend their accounts. But at the very least, if you can identify them, you can prevent them from getting decent analytics on how their posts are doing.

Kathy:
He’s definitely going to make security at Twitter… And that’s going to have sort of a trickle down like anything on Twitter. I mean, it’s sort of the behemoth of social media at this point of anything that is happening in the world. I always call it like earthquake Twitter. When I lived in California and there was an earthquake, it was like, okay, hit Twitter. Where was this? That was the first place we would look. It is such a touchpoint of what’s happening in the world, whether it be politics or an earthquake or anything else.

Kathy:
So having Mudge be the head of security there is definitely going to have an effect in how conversations are happening. In this article on Reuters, he definitely praised a recent change at Twitter where people are now being encouraged to add to the conversation rather than just re-tweeting something without providing some kind of commentary themselves. Those types of things, I think it’s going to be a good thing for Twitter. What do you think?

Ram:
I think it’s definitely going to be an improvement. Twitter is a large company that grew extremely quickly, so I’m sure that there are still things that are held together with duct tape and bubble gum in a few critical places. But having someone who’s used to working with that and finding those things is definitely going to go a ways towards getting them improved.

Kathy:
Definitely something to watch. Hey, I bet you that GO SMS has some bubble gum and toothpicks somewhere.

Ram:
Oh, okay. There’s Android app called GO SMS Pro, and I guess it’s installed on a hundred million phones.

Kathy:
Yeah.

Ram:
I want it to say sites for a second, because we’re always talking about this being installed on 5 million websites or a hundred thousand websites. This is not sites. This is phones.

Kathy:
And what’s going on?

Ram:
Android phones. This is actually really bad. If a user sent a media message to someone else that wasn’t using the app, it would generate a shortened URL linking to that media on their CDN. You just click the shortened link to view the image or the voice recording or the whatever. And this is something known as an IDOR, an insecure direct object reference, where you can just sort of go through a bunch of like ID equals one, ID equals two, ID equals three to see what’s related to each ID being referenced.

Ram:
In this case, the content was sequentially stored in hexadecimal format on their CDN. And basically it was possible to just go through all the links on the CDN to scoop up some pretty scary stuff. Things like photos of user’s cars, screenshots of other messages and Facebook posts, explicit photos, videos, audio recordings, and photos of sensitive documents. So like basically all this stuff you don’t really want to be public, period.

Kathy:
So it looks like this vulnerability was discovered by Trustwave and they disclosed the vulnerability on August 18th and did not receive a reply. After 90 days of their initial responsible disclosure to the GO SMS Pro developers, they did not receive anything back. So now it is a public… I wonder if anybody else other than the security researchers at Trustwave has discovered this and found anything sensitive.

Ram:
Here’s the thing, the longer you wait… Responsible disclosure is important, because you absolutely do want to give developers time to fix something. But the longer you wait, the bigger the chance of someone independently discovering this and exploiting this in the wild. And for something like this where it’s a pretty simple hack. You can just… I’m not going to tell you how to do it, but I’m fairly sure that pretty much everyone we work with could look at that article and go, “I know how to do this.”

Kathy:
Yeah, exactly.

Ram:
So something like that, it’s almost definitely being exploited in the wild already. And we still haven’t heard from them. No one has heard anything from them in 90 days. Uninstall that app, please, as soon as possible if you have it installed on your Android phone. Is it Android? Yeah. It’s Android.

Kathy:
Yeah, it’s just Android. A hundred million people are using this. If you are using GO SMS Pro, it’s time to stop. I don’t have this app. I don’t have Android, so I don’t know if there’s any way to delete your previous messages. But if there is, you might want to.

Ram:
Yeah. I mean, I use Signal. Telegram is supposed to be really good. But honestly, this is less secure than standard SMS or MMS. This is less secured than just plain old over the phone messages.

Kathy:
Right, and there are problems with SMS messages as it is. Signal. Signal is my favorite. Except every time I have a contact, like someone I haven’t even talked to in 10 years, but they’re still on my contacts list, it’s like, hey, this person that you probably don’t want to talk to, they’re now on Signal. Just thought we’d let you know.

Ram:
I’ve gotten so many of those. Oh man. It’s like now I know which of my friends are paranoid. But in the interest of paranoia, one of the downsides to just about any SMS application is that the built-in keyboard on your phone can read what you’re typing into it even if it’s end-to-end encrypted. If your phone does get infected with any kind of malware or anything like that, information leakage, then an end-to-end encrypted messaging app is not going to really help that much.

Kathy:
Right. Right. It’s just another reminder. Be careful with the apps you’re using, but also keep your phones updated. And if something is highly sensitive, just maybe don’t send things over the wires or the air.

Ram:
I mean, don’t necessarily shame people, but yeah, security breaches happen.

Kathy:
All the time.

Ram:
Way too often.

Kathy:
Just expect them and make your behavior or adjust your behavior accordingly, I suppose. So that’s all the news we have this week. We are hiring. Head over to Wordfence.com. Scroll to the bottom. See careers. If you are not on our mailing list, you might want to get on our mailing list. There is a link down there as well. Because whenever we find a vulnerability in WordPress, we make sure that our users are the first to know about that. There’s no cost to being on that mailing list to sign up.

Kathy:
We don’t send a lot of marketing emails at all, and we might have some stuff in the footer, but perhaps that would be good for you to know if you have a WordPress site. And of course, subscribe to the podcast as well while you’re over there if you want to get notified when we post a new podcast. That’s all we’ve got this week. Anything else, Ram?

Ram:
It’s been a pleasure as always.

Kathy:
It’s always a pleasure, isn’t it?

Ram:
It is.

Kathy:
And next week, I think we’re going to take next week off because it is Thanksgiving. So we will be back the week after that. Have a good one and thanks for listening.

You can find Wordfence on Twitter, Facebook, Instagram. You can also find us on YouTube, where we have our weekly Wordfence Live on Tuesdays at noon Eastern, 9:00 AM Pacific.