1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Is it really impossible to block majestic bot? Or would this work?

Discussion in 'Black Hat SEO' started by taketheride, Jun 8, 2015.

  1. taketheride

    taketheride Junior Member

    Joined:
    Mar 3, 2015
    Messages:
    143
    Likes Received:
    19
    I've been trying on several sites with different combinations of code to block majestic crawlers through htaccess, but it just doesn't work. I'm having no problem with the other bots like ahrefs and OSE. Don't want to leave a footprint by using neither the robots file, nor spyderspanker/other plugins right now. But I was thinking, wouldn't it be possible to block all bots except for those coming from search engines? I've been looking around and trying to learn how to put this into code, but can't seem to figure it out. So I'm wondering:

    - Do you think this would be possible without obstructing the bots you want to have crawling the site?

    - Is there a way to distinguish between bots and real people/browsers in htaccess?
     
  2. ghostlyman

    ghostlyman Newbie

    Joined:
    Jun 7, 2015
    Messages:
    15
    Likes Received:
    7
    Occupation:
    SEO
    Location:
    Denver, USA
    Home Page:
    Hey, I know you said that you don't want to use SpyderSpanker plugin but I talked to few people who have decent sized PBNs and they said that they have SS installed on all sites and there hasn't been any problems with their network yet. I don't think the plugin itself leaves the footprint. The footprint would be the fact that crawlers are blocked, plugin wouldn't be a footprint if that makes sense
     
    • Thanks Thanks x 1
  3. mikpel

    mikpel Regular Member

    Joined:
    Dec 1, 2012
    Messages:
    244
    Likes Received:
    120
    What code are you using to block majestic? Just add this to you .htaccess and majestic crawler gets blocked:
    Code:
    RewriteEngine On
    RewriteBase /
    RewriteCond %{HTTP_USER_AGENT} Majestic-12 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} Majestic-SEO [NC]
    RewriteRule .* - [F]
    
    There is no need to pay 39$ for such a simple task as blocking bots (spyder spanker). Just add as many unwanted user agents in your .htaccess as you want.
     
    • Thanks Thanks x 1
  4. taketheride

    taketheride Junior Member

    Joined:
    Mar 3, 2015
    Messages:
    143
    Likes Received:
    19
    That's similar to what I've been using, except I redirect it to another website. Don't think I included the user agent with "majestic-seo" before though, just mj12, will try that out.