18X Speedup in Wordfence Scan

Wordfence 6.2.0 was released yesterday and it includes something really special: a huge improvement in scan performance. I’m going to share with you what changed and why Wordfence 6.2.0 is the same great quality malware and security scan running up to 18X faster.

Malware scanning is a critical part of the layered approach to protection that Wordfence provides. The Wordfence Firewall stops attacks in their tracks. Scanning is how we determine if a hacker has gained access to your site. You’ll recall that we’ve also integrated the malware scan into the Wordfence Firewall to scan file uploads as they arrive. This gives you three layers of protection against attacks.

Our Firewall has always been extremely fast and the malware scan built into the firewall uses the same model of execution. While we are always looking for ways to optimize our firewall code, we really don’t need to improve in that area because it’s so darn fast already.

However, we had received reports from a few users that scanning would take quite a long time on larger sites. We have also had reports from a few hosting providers that when a large number of scans are running concurrently (200 or more), they could use a lot of disk IO resources.

So we decided to tackle this issue head-on and work closely with our customers and hosting providers.

Working to make scan faster

I met with one large host in person and had a very informative and collaborative discussion (Thanks Heidi!!) that included follow-up conversations with our team and a direct channel of communication with our devs and QA team. Then we put our heads down and got to work.

Ryan Britton, one of our senior developers at Wordfence led the technical effort and Matt Rusnak led the QA effort on this project. They went away and worked on this and when they came back with the numbers, I couldn’t believe the results.

To measure performance, we isolated the process running Wordfence and monitored disk reads. The above bar charts show the amount of data read in megabytes for the older version of Wordfence compared to 6.2.0 which is in green. Less is better because it reduces the amount of IO load on the physical server hosting the site running Wordfence.

• For Wordfence 6.2.0 we halved the IO load on sites running a brand new plain WordPress installation.
• On a small site of around 92MB we achieved a 3.6X reduction in disk IO load.
• On a larger site with 682MB we are seeing an impressive 18.4X reduction in disk IO load.

Larger sites will see an even greater increase in speed and reduction in IO load during scans. We don’t know what the average size of a WordPress site is, but we estimate that it’s similar to the large site model we used. My personal blog, for example, is 552MB of files and it’s been around for a reasonable amount of time.

Scan functionality and detection capability stays exactly the same.

We haven’t reduced the number of signatures we’re using for detection or the number of files we read. There was a bugfix included in this release where we no longer scan .txt files unless you have ‘High Sensitivity’ scanning enabled, but that is unrelated to performance improvements.

How we made these huge performance gains

We achieved this huge speed increase by changing the way we’re performing MD5 and SHA256 hashes of files, getting smarter about not hashing files where we aren’t using the hash and reducing the hashing operation to a single streaming read on each file.

We also removed counters that were not needed that were increasing file reads. In addition, this new release keeps better track of which directories have been scanned so that we don’t need to reiterate through a large list of directories when the scan resumes.

It’s worth noting that image-heavy sites will see a particularly large improvement in scan speed.

…And then we went even further!

One of the amazing things about the Wordfence team is that they are super passionate about what they do. For 6.2.0 I was expecting to get a modest increase in scan speed. Instead we got a huge gain in scan speed. Then the customer service team got involved and encouraged Dev and QA to go even further.

What they came up with is what we’re calling a “Low resource scan”. When this option is enabled, Wordfence does exactly the same amount of work, but it introduces an artificial delay in the scan which consumes no additional resources but spreads the disk IO load out over time. This reduces peak usage load on servers that have slow disk or that are hosting a lot of other busy websites.

You can enable the “Low resource scan” by going to the Wordfence options page on your WordPress site, scroll down to the “Scans to Include” heading and then at the bottom of that section you can find the option labeled “Use low resource scanning. Reduces server load by lengthening the scan duration.”. You can find the documentation for that option here.

As you can tell from above, we have massively reduced the amount of disk IO that Wordfence generates, but just in case that’s not enough and your hosting provider asks you to reduce things further, you can now enable this option.

In our tests, the low resource scan takes 1.7 to 3.7 times longer to run, depending on your site, while reducing peak load. Again, it does the same amount of work, just spread over more time.

The low resource scan combined with our much faster scan engine gives you a very low impact way to regularly scan for malware while keeping your hosting provider happy and your site super fast. We recommend most people leave this setting off, but in extreme cases of limited resources, customers can now enable this setting.

We recommend the following configuration:

You will find the best performance gains when Wordfence is using the default scan configuration. For our own sites we leave low-resource scanning off and just use the default Wordfence options. The scan is now incredibly fast.

On my own blog with 6127 files totaling 552MB of space, my scan now finishes in 24 seconds from a cold start to finish. I run my personal site on the cheapest $10/month Linode server available. That is incredibly fast on a very modest server.

We think the standard configuration will work great for the vast majority of sites out there.

In Conclusion

As always a huge thanks from me to our incredible team for this collaborative effort. And a very special thanks to the customers and hosting providers who worked with us to help us better understand their needs.

Wordfence remains committed to securing the WordPress community and platform. To that end we work closely with our customers and hosting providers to ensure that we continue to provide the best protection available for WordPress. This is another major leap forward in achieving our goal of a secure WordPress community and a more secure Web.

As always we welcome your feedback in the comments below, or you can contact us using one of the channels on our contact page.