loknath.tcms
Newbie
- May 18, 2013
- 40
- 2
How to block Wordpress site from major crawling like ahref moz majestic etc. permanenty
Thanks
Thanks
How to block crawling
- Thread starter loknath.tcms
- Start date
-
- Tags
- back hat seo linkbuildilng seo
what_hat?
Regular Member
- Sep 1, 2012
- 396
- 229
You can use a plugin: http://spyderspanker.com/ https://wordpress.org/plugins/spiderblocker/
Or you could block SEO crawlers using your .htaccess file: https://www.blackhatworld.com/seo/pbn-htaccess-question.902210/
Or you could block SEO crawlers using your .htaccess file: https://www.blackhatworld.com/seo/pbn-htaccess-question.902210/
530inc
Regular Member
- Dec 18, 2013
- 201
- 77
I usually put this in my htaccess for my wordpress sites:
Code:
# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} .*Twice.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} .*Yand.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} .*Yahoo.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} .*Voil.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} .*libw.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} .*Java.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} .*Sogou.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} .*psbot.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} .*Exabot.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} .*boitho.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} .*ajSitemap.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} .*Rankivabot.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} .*DBLBot.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} .*MJ1.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} .*Rankivabot.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} .*ask.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} .*rogerbot.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} .*exabot.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} .*MJ12bot.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} .*dotbot.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} .*gigabot.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} .*SemrushBot.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} .*BLEXBot.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} .*xenu.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} .*AhrefsBot.* [NC]
RewriteRule ^.* - [F,L]
Order Allow,Deny
Allow from all
Deny from 110.0.0.0/8
Deny from 111.0.0.0/8
Deny from 112.0.0.0/5
Deny from 120.0.0.0/6
Deny from 124.0.0.0/8
Deny from 125.0.0.0/8
Deny from 147.0.0.0/8
Deny from 169.208.0.0
Deny from 175.0.0.0/8
Deny from 180.0.0.0/8
Deny from 182.0.0.0/8
Deny from 183.0.0.0/8
Deny from 202.0.0.0/8
Deny from 203.0.0.0/8
Deny from 210.0.0.0/8
Deny from 211.0.0.0/8
Deny from 218.0.0.0/8
Deny from 219.0.0.0/8
Deny from 220.0.0.0/8
Deny from 221.0.0.0/8
Deny from 222.0.0.0/8
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPress