IP Blacklist Update: The Launch and Evolution of The Wordfence IP Blacklist

One of our passion projects at Wordfence has been to find a way to create and run an IP blacklist. We have known for a long time which IPs are attacking the sites we protect and that if we can block those IPs outright, it would be a powerful way to improve the security we provide to our customers.

Blocking a bad IP completely is more effective and safer than just blocking its malicious requests, because you don’t allow it to gather information about the target website it is about to attack.

We launched the Wordfence Premium IP Blacklist 1 month ago and it has been incredibly effective at improving the security of the websites that Wordfence protects.

Today I want to share the story behind why we decided to create the Worfence Premium IP Blacklist, how we created and launched it, how it has evolved rapidly to contain over 4,000 dynamically updated IPs and how quickly it is growing.

Creating and Launching the IP Blacklist

When we conceived of the IP blacklist about a year ago the challenge we had was to have a way to block bad IPs accessing our customer sites and let the good guys through without actually distributing a list of bad IPs to customer sites. We wanted to avoid distributing the list so that we didn’t also provide other attackers with a list of possibly compromised targets to attack.

The trouble with not distributing a list of bad IPs is that the obvious alternative is to have our customer sites look-up whether or not a visitor IP is bad for every request. That would have awful performance implications. It would mean a visitor to a site has to wait until the lookup has completed until their page loads. Yuck!

After background-processing this problem for several months our team came up with a way of getting the best of both worlds. We realized that we could distribute the list of IPs as “hash prefixes” which achieve both of our goals. Our customer sites would not be slowed down because 99.9% of good requests would not need to do a lookup on Wordfence servers. It also made it very difficult for attackers to reverse engineer the IP blacklist to reveal new targets.

Blocking baddies without blocking goodies

Once we had figured out how to implement the list, the dev team started work on it and once complete the feature spent a long time with our quality assurance team. We wanted to do a great job of protecting customer sites while creating zero interference with their legitimate traffic.

First, we developed a way for anyone blocked by the list to report false positives. Since launch, the number of false positives (good people being blocked accidentally) reported is  very low, which is impressive when you consider that the IP blacklist has processed hundreds of millions of requests since launch.

Then we took a “phased” approach to the launch.

Blacklist Phase 1:

We launched the blacklist with a small number of 100 IPs on the list. Then we gradually added around 100 additional IPs by hand every day or two, until we hit 1000 IPs. We constantly evaluated how the list was doing and if there were any false positive reports. There were none.

Blacklist Phase 2: 

Once we hit that point, we had not received any false positive reports, so we switched to “dynamic” mode where the list auto-updated hourly. The algorithms we used in dynamic mode phase 2 to continuously identify bad IPs were “coarse” and only included the worst offenders and excluded some IPs that are engaging in lower frequency attacks.

Blacklist Phase 3:

In phase 3 we started optimizing our algorithms and lowering the ‘bar’ for what we considered a bad IP. We did this gradually and are still in this phase as we continue to increase the sensitivity of our algorithms to detect malicious IPs. The list is now fully live and is blocking a large number of requests from malicious IPs on our Premium customer sites every day.

The Growth of the Wordfence IP Blacklist

The graph below does a great job of illustrating the phases described above and the gradual way we have grown the IP blacklist and how it continues to grow:

You can see how we steadily added to the IP blacklist manually until it hit 1000. This took about a week. Then we switched to dynamic mode and you begin to see the fluctuations in the number of IPs on the list as the number of attackers changes over time.

As we increased the sensitivity of our algorithms there is a steady increase in the number of IPs on the list. We are currently at around 4,500 IPs on the list and we continue to increase sensitivity.

Changes Over Time: The Blacklist is a Living Breathing Thing

The Wordfence blacklist isn’t something we just continually add to. It is a living breathing thing which is updated continuously. IPs are continuously added and removed. You’ll notice a dip in the chart above which shows how we gradually removed over 1,000 IPs over a 1 week period as they stopped behaving maliciously.

IPs that start attacking are immediately added to the blacklist. IPs that stop attacking are removed over time and that time is relatively short – usually a few days, depending on their behavior.

In the chart below we show how dynamic the list is over a single 24 hour period. We chose the 11th of April for our example. You can clearly see that during the day we are constantly adding malicious IPs as they become active and constantly removing IPs that are no longer behaving badly.

I know that many of you are “active managers” of your WordPress site security. It can begin to feel like a video game as you’re monitoring live traffic, adding IPs to your block list and blocking networks or user-agents. The blacklist solves that problem comprehensively. It relies on an incredible amount of global attack data and does an excellent job of blocking far more malicious IPs that any administrator could manage. It also unblocks IPs as they stop behaving badly to ensure that false positives don’t occur.

The Wordfence Premium blacklist blocks new malicious IPs very quickly and any IPs that stop behaving maliciously are quickly unblocked. From the perspective of a WordPress site owner, in my opinion, it is a site admin’s dream come true.

Ranking Signals for IP Reputation

We think about IP reputation much the same way as Google thinks about their search algorithm. We use what we refer to as ‘ranking signals’ which are factors that influence how malicious we consider an IP address to be.

We have no plans to publicly disclose all the ranking signals that we use for IP reputation. We don’t want attackers to be able to completely reverse engineer our methods. But as we grow the list to a significant size, we may make suggestions for network owners to help them ensure that the IPs on their network retain a healthy ranking.

For example, if you are a hosting provider, a compromised website or hosting account can be used to attack other sites and online targets. That would negatively affect your IP’s ranking on blacklists. Responding quickly to a compromised account or website will ensure that the site IP is not blacklisted and other accounts on the same system are not negatively impacted.

Next Steps: Optimizing Current Signals, Developing New Signals

This week we wrote about thousands of home routers that are being used to attack WordPress websites that Wordfence protects. These home routers are engaging in very low volume attacks for short periods of time. This is a new signal that we have discovered and is a relatively weak signal. But when combined with other signals, it becomes strong and clearly reveals IPs that belong on our blacklist.

We are constantly engaged in R&D like the research we performed to uncover the hacked home routers. This helps us discover and implement new ranking signals to grow and optimize our IP blacklist.  Now that the blacklist is live and dynamic, our goal is to continuously improve, measure and evaluate our algorithms going forward.

The real-time IP blacklist is one of many great security features in Wordfence Premium. If you aren’t already using Wordfence, try out our completely free (and awesome) plugin today. Then when you are ready to upgrade to Premium, you can visit our home page on wordfence.com to find out more.