1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Blocking Crawlers

Discussion in 'Black Hat SEO' started by InsanelySane, Apr 22, 2014.

  1. InsanelySane

    InsanelySane Power Member

    Joined:
    Nov 23, 2013
    Messages:
    562
    Likes Received:
    111
    How many people are doing this for their sites? Im only interested in blocking via robots.txt. However, I've heard ahrefs and majestic tend to not honor your decision.
     
  2. BuildMoreLinks

    BuildMoreLinks Jr. VIP Jr. VIP Premium Member

    Joined:
    Jun 7, 2012
    Messages:
    1,916
    Likes Received:
    655
    Location:
    17.3660° N, 78.4760° E
    This is from Ahrefs FAQ page:


    The quick way to prevent AhrefsBot visiting your site is
    to put these two lines into the /robots.txt file on your server:


    The problem with AHrefs is that it doesn't need to spider your site to get the data - it only needs to spider the sites that link to you and they can construct an inbound linking profile. By blocking them you are only stopping them from spidering links from your site to others.
     
    • Thanks Thanks x 1