blocking the seo bots

Discussion in 'White Hat SEO' started by W9go, Feb 20, 2017.

Tags:
  1. W9go

    W9go Jr. VIP Jr. VIP

    Joined:
    May 16, 2011
    Messages:
    5,672
    Likes Received:
    1,212
    Gender:
    Male
    Occupation:
    chasing girls
    Location:
    chasing girls
    someone can give me the code for ht access to block the seo bots crawling my page ...

    thanks
     
  2. Linus persson

    Linus persson Registered Member

    Joined:
    Jan 15, 2017
    Messages:
    68
    Likes Received:
    22
    Gender:
    Male
    Location:
    Sweden
    <html>
    <head>
    <title>...</title>
    <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
    </head>
     
  3. Nut-Nights

    Nut-Nights Jr. VIP Jr. VIP

    Joined:
    Jun 20, 2013
    Messages:
    6,798
    Likes Received:
    4,088
    Location:
    Hell
    Home Page:
    • Thanks Thanks x 1
  4. Linus persson

    Linus persson Registered Member

    Joined:
    Jan 15, 2017
    Messages:
    68
    Likes Received:
    22
    Gender:
    Male
    Location:
    Sweden
  5. Google Prince

    Google Prince Jr. VIP Jr. VIP

    Joined:
    Dec 24, 2015
    Messages:
    184
    Likes Received:
    138
    Location:
    Google's Search Engine
    Here ya go buddy ;)

    Code:
    SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
    SetEnvIfNoCase User-Agent .*exabot.* bad_bot
    SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
    SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
    SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
    SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
    SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
    <Limit GET POST HEAD>
    Order Allow,Deny
    Allow from all
    Deny from env=bad_bot
    </Limit>
     
    • Thanks Thanks x 2
  6. issorc

    issorc Power Member

    Joined:
    Feb 20, 2016
    Messages:
    767
    Likes Received:
    662
    Gender:
    Male
    He wants to block crawlers like Ahrefs and Majestic and they don't give a fuck whether you'd like your site to be indexed or not.
     
  7. Gromit

    Gromit Regular Member

    Joined:
    Feb 17, 2014
    Messages:
    439
    Likes Received:
    134
    Location:
    UK
    Pretty sure there's a big list floating around for this on here.

    Have a search on the forum for some of the bot names posted above.
     
  8. Werombi

    Werombi Newbie

    Joined:
    May 26, 2015
    Messages:
    38
    Likes Received:
    18
    Occupation:
    MD of catering and recruiter
    Location:
    Australia
    I know this post few months old though would like to know whats the difference with the quoted htaccess and the below?

    RewriteEngine on
    RewriteCond %{HTTP_USER_AGENT} ^USER_AGENT [NC,OR]
    RewriteRule ^.* - [F,L]

    I use wordpress and was given a list of agents to block wwith the above conditions. If 1 better over another?