How referrer spam affects search engine rankings

A question I receive fairly often is: Wordfence ran a scan on my site and found a known malicious URL in one of my files. The file is a backup of my database or a log file. Where does this URL come from, what is it’s impact and what should I do about it?

I’d like to explain this in detail because it will help you understand referrer spam and what you see in your real-time traffic.

A common tactic on the Net that malicious webmasters use is to try and publicize their site by engaging in referrer spam. What this means is that they will create an automated web crawler much like Google’s Googlebot. They then send this crawler out to visit hundreds of millions of websites and the crawler pretends to be a normal web browser. The user-agent string (identification) that the crawler sends is Chrome or Internet Explorer or another human looking web browser. But the important difference is that it sends a fake referrer string to any website it visits. That means that it tells the logging application on any website it visits that it arrived from a website that it didn’t actually arrive from. Lets call that site example.com.

The effect is that there appears to be a fake person going around visiting a hundred million websites and arriving at those websites from someone else’s website.

You and I both enjoy looking at which websites are sending us traffic. When the bad crawler arrives at our site and visits a few hundred pages, our logging program will log that it came from example.com and of course we’re going to get very excited about going to visit example.com when our logging program tells us that example.com is sending our site all this new traffic. So we visit the site….

If it’s a phishing site we might get fooled into giving away some sensitive information. If it’s a hacking site we might end up getting our web browser hacked and malware installed on our machine.

If you see a new website in your list of referrers in your logging application, visit it using Chrome which uses Google’s safe browsing list to block bad sites. Or you can type the URL into this box and hit the button to get a report on it:

Referrer spam has other effects. Occasionally referrers are displayed publicly on websites or they accidentally end up in logfiles that are visible to the public and to Googlebot. That sends example.com (the malicious website) even more traffic because it gets a boost in the search engine rankings thanks to the new backlinks and it gets more visitors who see the URL appearing on some random web page and wonder what it is and pay it a visit.

So that is what referrer spam is and why people engage in it, particularly malicious websites.

When our WordPress administrators do a scan with Wordfence, we look for malicious URL’s in all your files. Occasionally we’ll find one in a log file that we scanned or in a backup of your database. I receive emails from admins from time to time thinking that they’ve been hacked when a malicious URL shows up in a file. It’s quite normal to find malicious URL’s in your log files or database backups because unfortunately referrer spam is commonplace. The danger comes from accidentally storing those log files or DB backups in a directory that is publicly accessible. You must NEVER do this. Not only will your sensitive data become public, but Google will start indexing that file with all those malicious URL’s and you may incur a severe search engine ranking penalty. This sounds like obvious advice, but you’d be appalled at how many people install a backup plugin or logging application and use the default settings which inadvertently delivers backups or log files into a public directory.

So in conclusion:

• Referrer spam has sadly become commonplace and you will eventually find a malicious URL appearing in your database or in a log file.
• As long as that log file or database record is not visible to the public and to crawlers, it won’t affect your search engine rankings.
• Be careful which sites you visit when viewing your logs and referring websites.
• Store your log files and database backup files in a secure private directory.