1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

blocking the seo bots

Discussion in 'White Hat SEO' started by W9go, Feb 20, 2017.

Tags:
  1. W9go

    W9go Jr. VIP Jr. VIP Premium Member

    Joined:
    May 16, 2011
    Messages:
    4,884
    Likes Received:
    1,002
    Gender:
    Male
    Occupation:
    chasing girls
    Location:
    chasing girls
    someone can give me the code for ht access to block the seo bots crawling my page ...

    thanks
     
  2. Linus persson

    Linus persson Registered Member

    Joined:
    Jan 15, 2017
    Messages:
    68
    Likes Received:
    22
    Gender:
    Male
    Location:
    Sweden
    <html>
    <head>
    <title>...</title>
    <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
    </head>
     
  3. Nut-Nights

    Nut-Nights Jr. VIP Jr. VIP

    Joined:
    Jun 20, 2013
    Messages:
    5,296
    Likes Received:
    3,375
    Location:
    Hell
    Home Page:
    • Thanks Thanks x 1
  4. Linus persson

    Linus persson Registered Member

    Joined:
    Jan 15, 2017
    Messages:
    68
    Likes Received:
    22
    Gender:
    Male
    Location:
    Sweden
  5. Google Prince

    Google Prince Jr. VIP Jr. VIP

    Joined:
    Dec 24, 2015
    Messages:
    162
    Likes Received:
    93
    Location:
    Google's Search Engine
    Here ya go buddy ;)

    Code:
    SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
    SetEnvIfNoCase User-Agent .*exabot.* bad_bot
    SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
    SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
    SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
    SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
    SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
    <Limit GET POST HEAD>
    Order Allow,Deny
    Allow from all
    Deny from env=bad_bot
    </Limit>
     
    • Thanks Thanks x 2
  6. issorc

    issorc Jr. VIP Jr. VIP

    Joined:
    Feb 20, 2016
    Messages:
    688
    Likes Received:
    528
    Gender:
    Male
    He wants to block crawlers like Ahrefs and Majestic and they don't give a fuck whether you'd like your site to be indexed or not.
     
  7. Gromit

    Gromit Regular Member

    Joined:
    Feb 17, 2014
    Messages:
    267
    Likes Received:
    91
    Location:
    UK
    Pretty sure there's a big list floating around for this on here.

    Have a search on the forum for some of the bot names posted above.
     
  8. Werombi

    Werombi Newbie

    Joined:
    May 26, 2015
    Messages:
    34
    Likes Received:
    14
    Occupation:
    MD of catering and recruiter
    Location:
    Australia
    I know this post few months old though would like to know whats the difference with the quoted htaccess and the below?

    RewriteEngine on
    RewriteCond %{HTTP_USER_AGENT} ^USER_AGENT [NC,OR]
    RewriteRule ^.* - [F,L]

    I use wordpress and was given a list of agents to block wwith the above conditions. If 1 better over another?