1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Looking for updated htaccess code to block crawlers

Discussion in 'Black Hat SEO' started by Openside, Aug 4, 2016.

  1. Openside

    Openside Power Member

    Joined:
    Dec 16, 2013
    Messages:
    543
    Likes Received:
    95
    Home Page:
  2. cnick79

    cnick79 Jr. VIP Jr. VIP

    Joined:
    Jun 10, 2010
    Messages:
    691
    Likes Received:
    369
    Location:
    Google's SandBox
    What bots are getting through? Why not add them to your list?

    Also, htaccess and robots file are not sure fire ways to stop bots. It's "best practice" to identify your not but nothing can stop a scripted bot from accessing your site unless you block by IP/ IP range.
     
  3. pauverinox

    pauverinox Newbie

    Joined:
    Apr 21, 2016
    Messages:
    18
    Likes Received:
    1
    Got this from a thread here.

    Code:
    # BEGIN WordPress
    <IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteBase /
    RewriteRule ^index\.php$ - [L]
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteRule . /index.php [L]
    </IfModule>
    
    # END WordPress
    
    
    SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
    SetEnvIfNoCase User-Agent .*exabot.* bad_bot
    SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
    SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
    SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
    SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
    SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
    SetEnvIfNoCase User-Agent .*semrushbot.* bad_bot
    SetEnvIfNoCase User-Agent .*ia_archiver.* bad_bot
    SetEnvIfNoCase User-Agent .*searchmetricsbot.* bad_bot
    SetEnvIfNoCase User-Agent .*seokicks-robot.* bad_bot
    SetEnvIfNoCase User-Agent .*sistrix.* bad_bot
    SetEnvIfNoCase User-Agent .*lipperhey spider.* bad_bot
    SetEnvIfNoCase User-Agent .*ncbot.* bad_bot
    SetEnvIfNoCase User-Agent .*backlinkcrawler.* bad_bot
    SetEnvIfNoCase User-Agent .*archive.org_bot.* bad_bot
    SetEnvIfNoCase User-Agent .*meanpathbot.* bad_bot
    SetEnvIfNoCase User-Agent .*pagesinventory.* bad_bot
    SetEnvIfNoCase User-Agent .*aboundexbot.* bad_bot
    SetEnvIfNoCase User-Agent .*spbot.* bad_bot
    SetEnvIfNoCase User-Agent .*linkdexbot.* bad_bot
    SetEnvIfNoCase User-Agent .*nutch.* bad_bot
    SetEnvIfNoCase User-Agent .*blexbot.* bad_bot
    SetEnvIfNoCase User-Agent .*ezooms.* bad_bot
    SetEnvIfNoCase User-Agent .*scoutjet.* bad_bot
    SetEnvIfNoCase User-Agent .*majestic-12.* bad_bot
    SetEnvIfNoCase User-Agent .*majestic-seo.* bad_bot
    SetEnvIfNoCase User-Agent .*dsearch.* bad_bot
    SetEnvIfNoCase User-Agent .*blekkobo.* bad_bot
    SetEnvIfNoCase User-Agent .*screaming frog seo spider/*.* bad_bot
    SetEnvIfNoCase User-Agent .*PHPCrawl.* bad_bot
    SetEnvIfNoCase User-Agent .*gocrawl.* bad_bot
    SetEnvIfNoCase User-Agent .*DigExt.* bad_bot
    SetEnvIfNoCase User-Agent .*DomainSONOCrawler.* bad_bot
    SetEnvIfNoCase User-Agent .*TweetmemeBot.* bad_bot
    SetEnvIfNoCase User-Agent .*OpenHoseBot/2.1.* bad_bot
    SetEnvIfNoCase User-Agent .*Kraken/0.1.* bad_bot
    SetEnvIfNoCase User-Agent .*-Java-.* bad_bot
    SetEnvIfNoCase User-Agent .*ubermetrics.* bad_bot
    SetEnvIfNoCase User-Agent .*best-seo.* bad_bot
    SetEnvIfNoCase User-Agent .*Synapse.* bad_bot
    SetEnvIfNoCase User-Agent .*Harvest.* bad_bot
    SetEnvIfNoCase User-Agent .*Harvester.* bad_bot
    SetEnvIfNoCase User-Agent .*harvester.* bad_bot
    SetEnvIfNoCase User-Agent .*harvest.* bad_bot
    
    <Limit GET POST HEAD>
    
    Order Allow,Deny
    
    Allow from all
    
    Deny from env=bad_bot
    
    </Limit>
     
  4. StSignal

    StSignal Junior Member

    Joined:
    Jun 15, 2016
    Messages:
    137
    Likes Received:
    11
    Gender:
    Male
    Occupation:
    SEO & AdSense
    Location:
    Ukraine
    Exactly,nothing can stop it after activation.
     
  5. Openside

    Openside Power Member

    Joined:
    Dec 16, 2013
    Messages:
    543
    Likes Received:
    95
    Home Page:
    I have pretty much the same code as above...just weird that all of a sudden everything is showing when 6 months ago nothing was showing up!