How to Block Referrer Spam Bots

it-security-01About referrer spam

Referrer spam requests are requests for some pages with a faked referrer string (where the user came from). Normally this is just annoying as it appears in your webstats. More info about referer spam you can read at wikipedia

But sites could be hit by many requests within short period of time (seconds) affecting server performance. The requests usually come from very different IP addresses, so blocking with iptables is not an option. For example, I’ve noticed many nasty activities including domains make-money-online.7makemoneyonline.com and buttons-for-website.com in the last few days but requests have came from hundreds IP addresses and many different countries.

But I wanted to block only these referrer domains (among many others) not whole countries and IPs.

So what to do?

The trick here is to block not IP addresses but http referrers instead. This tutorial is based on Apache web server and .htaccess file. There are two methods you can use: simpler, with .htaccess file only and another one with .htaccess file calling a file with black-listed domains.

1 method: Blocking referrer spam bots with htaccess only

1) Add into .htaccess:

RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_REFERER} domain1
RewriteRule ^.* - [F]
RewriteCond %{HTTP_REFERER} domain2
RewriteRule ^.* - [F]
RewriteCond %{HTTP_REFERER} domain3
RewriteRule ^.* - [F]

Explanation

If there is a referer in the request and on of the words domain1, domain2 or domain3 are in the referrer, send them a 403 forbidden message. This way the server blocks the request early in the processing before any cgi-script is called. This saves CPU resources and bandwidth.

2 method: Blocking referrer spam bots with blacklist file

This method is useful when you blacklist increases too many nasty domains and it uses separate file just for keeping all of these bad sites.

1) Create file /etc/apache2/blacklist.txt and add:

domain1 -
domain2 -
domain3 -

The – at the end of the lines is important, but could be any character. If you use another character, modify the RewriteCond shown later.

2) Tell apache to load this file by adding the following to your /etc/apache2/apache.conf file:

Rewritemap refhashmap txt:/etc/apache2/blacklist.txt

3) Add the following for every virtual host you want to have the filter running:

RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_REFERER} ^http://([^/]+)
RewriteCond ${refhashmap:%1} ^-$
RewriteRule ^.* - [F]

If you already have the RewriteEngine and RewriteBase lines, you don’t need to repeat them. If you changed the char in the blacklist file above, set the char in the RewriteCond ${refhashmap:%1} line into the ^-$ block.

This way you have one site-wide blacklist and cleaner configuration files. Bye bye referrer spammers!!!

Share This Post

Recent Articles

Leave a Reply

 
© 2017 LoneShooter.com. All rights reserved. Site Admin · Entries RSS · Comments RSS