1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How to stop majestic and ahref to crawl your site ?

Discussion in 'Black Hat SEO' started by danhoff, Dec 10, 2012.

  1. danhoff

    danhoff Power Member

    Joined:
    May 11, 2010
    Messages:
    767
    Likes Received:
    55
    How can I stop majestic and ahref to crawl your site ? I don't want my sites to be in majestic and shref index.
     
  2. cgimaster

    cgimaster Power Member

    Joined:
    Jun 30, 2012
    Messages:
    525
    Likes Received:
    311
    Gender:
    Male
    Have you tried what they mention on their FAQ ? http://ahrefs.com/faqs.php


    • What is the name of your crawler, and how to block it via robots.txt?
      Remove my site from Ahrefs!


      - We use data gathered by AhrefsBot web crawler.
      If you ever decide to block it - see instructions at ahrefs.com/robot/ (consider dropping a note to us if you think there is a problem with the bot).
      - Remove my site from Ahrefs!
      When you block out bot via robots.txt, we stop crawling the site, but we continue finding and showing links pointing to this site from other sites.


    The quick way to prevent AhrefsBot visiting your site is
    to put these two lines into the /robots.txt file on your server:
    Code:
    user-agent: AhrefsBot
    user-agent: MJ12bot
    disallow: /
     
    • Thanks Thanks x 3
    Last edited: Dec 10, 2012
  3. seorank

    seorank Registered Member

    Joined:
    Nov 3, 2012
    Messages:
    54
    Likes Received:
    30
    dAmn he beat me to it, but yeah its simple robots.txt
     
    • Thanks Thanks x 1
  4. hnmfvnxe

    hnmfvnxe Regular Member

    Joined:
    Nov 10, 2012
    Messages:
    281
    Likes Received:
    78
    block with htaccess instead of robots
     
    • Thanks Thanks x 1
  5. notyourmatey

    notyourmatey Power Member

    Joined:
    Jul 7, 2011
    Messages:
    573
    Likes Received:
    324
    Location:
    TOP3
    you might want to look here:

    Code:
    http://www.blackhatworld.com/blackhat-seo/black-hat-seo/497725-method-how-stop-competitors-spying-your-links.html
     
  6. TheLinkGuy

    TheLinkGuy Power Member

    Joined:
    Nov 20, 2010
    Messages:
    627
    Likes Received:
    128
    Location:
    EU
    This is a great thread, reminded me to do that on a little network I will be building. I don't want any type of unnecessary exposure. Maybe someone could compile a list of robots.txt for disallowing all popular backlink checkers?
     
  7. SonaBG

    SonaBG Junior Member

    Joined:
    Feb 17, 2014
    Messages:
    180
    Likes Received:
    18
    Gender:
    Female
    Occupation:
    SEO Consultant
    Location:
    Armenia
    Home Page:
    but why you want to block?
    Ahrefs doesnt do anything bad