1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How to block robots in htaccess - NEED HELP!!

Discussion in 'Black Hat SEO' started by hangel, Dec 10, 2015.

  1. hangel

    hangel Newbie

    Joined:
    Jun 17, 2015
    Messages:
    16
    Likes Received:
    3
    I need to block the robots in .htaccess so that I don't have to use a plugin like spider spanker on the PBN domains. I want to block: majestic, ahrefs, opensite explorer, semrush, semalt as the main ones.

    I've checked other sources and I found this:
    htaccess

    SetEnvIfNoCase User-Agent .rogerbot. dangerbot
    SetEnvIfNoCase User-Agent .mj12bot. dangerbot
    SetEnvIfNoCase User-Agent .ahrefsbot. dangerbot
    SetEnvIfNoCase User-Agent .semrushbot. dangerbot

    Order Allow,Deny
    Allow from all
    Deny from env=dangerbot




    Robots.txt

    User-agent: AhrefsBot
    Disallow: /
    User-agent: MJ12bot
    Disallow: /
    User-agent: rogerbot
    Disallow:/
    User-agent: SemrushBot
    Disallow: /


    Can anyone tell me on how to do this exactly and how can I test spoof as a different user agent and robots to confirm it blocks robots? Thank you, I appreciate any thoughts!
     
  2. Penumbra

    Penumbra Power Member

    Joined:
    Apr 23, 2014
    Messages:
    742
    Likes Received:
    225
    Location:
    Antarctica
    • Thanks Thanks x 3
  3. hangel

    hangel Newbie

    Joined:
    Jun 17, 2015
    Messages:
    16
    Likes Received:
    3
    Awesome, thanks! With my next question, do you have any idea how I can test spoof as a different user agent and robots to confirm it blocks robots?
     
  4. TZ2011

    TZ2011 Senior Member

    Joined:
    Jun 26, 2011
    Messages:
    833
    Likes Received:
    864
    You can install addon for Firefox, "User Agent Switcher". Addon have a lot of standard user-agent strings, plus you can add custom strings manually.
     
    • Thanks Thanks x 1
  5. hangel

    hangel Newbie

    Joined:
    Jun 17, 2015
    Messages:
    16
    Likes Received:
    3
    Thanks TZ2011! I've installed the firefox add-on User Agent Switcher. I have also added custom strings manually as suggested. How do I exactly confirm that it has blocked those bots? Do I need to go to Majestic, Ahrefs, Semrush, etc to see if it still crawls my links? Thanks.
     
    Last edited: Dec 10, 2015
  6. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,489
    Likes Received:
    11,189
    Occupation:
    CHEAP
    Location:
    DATASETS
    Home Page:
    Some of these bastard link tools are using fake user agents. I had to start blocking by IP range too.
     
    • Thanks Thanks x 2
  7. bmills

    bmills Junior Member

    Joined:
    Jan 14, 2010
    Messages:
    148
    Likes Received:
    80
    Location:
    Southern California
    Home Page:
    Yeah tools are now just randomizing their user-agent...I mean why wouldn't they if they want to be the most accurate tool?
     
  8. TZ2011

    TZ2011 Senior Member

    Joined:
    Jun 26, 2011
    Messages:
    833
    Likes Received:
    864
    set blocking of those bots in the .htaccess, you will see if they are blocked if you visit the site with custom user agents that are faking seo bots. if everything is allright, you should get 403 or 404 page, depend of your htaccess setup.
     
  9. TZ2011

    TZ2011 Senior Member

    Joined:
    Jun 26, 2011
    Messages:
    833
    Likes Received:
    864
    @hangel set blocking of those bots in the .htaccess, you will see if they are blocked if you visit the site with custom user agents that are faking seo bots. if everything is allright, you should get 403 or 404 page, depend of your htaccess setup.
     
    • Thanks Thanks x 1
  10. hangel

    hangel Newbie

    Joined:
    Jun 17, 2015
    Messages:
    16
    Likes Received:
    3
    Awesome! Thanks for your help TZ2011!