1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How to Block 3rd Party Bots like Ahrefs, Majestic, SEMrush?

Discussion in 'White Hat SEO' started by dragiia, Oct 22, 2015.

  1. dragiia

    dragiia BANNED BANNED

    Joined:
    Apr 28, 2014
    Messages:
    125
    Likes Received:
    116
    I know you can change your robots.txt file to:

    User-agent: AhrefsBot
    Disallow: /

    However my site is still found like nothing has changed while my competitor's site is nowhere to be found. Am I doing something wrong or is there something I'm missing?
     
  2. cloudyseo

    cloudyseo Junior Member

    Joined:
    Feb 11, 2015
    Messages:
    164
    Likes Received:
    68
    Occupation:
    SEO
  3. dragiia

    dragiia BANNED BANNED

    Joined:
    Apr 28, 2014
    Messages:
    125
    Likes Received:
    116
    Obviously I Googled it you prick, that is is why I posted the common answer, Disallow: /, which didn't work. It's also why I asked if I was doing something wrong or missing something as the common popular answer didn't seem to work.

    http://lmgtfy.com/?q=Fuck+you
     
    • Thanks Thanks x 1
  4. cloudyseo

    cloudyseo Junior Member

    Joined:
    Feb 11, 2015
    Messages:
    164
    Likes Received:
    68
    Occupation:
    SEO
    If you would read carefully through the threads you will understand how ahrefs/majestic crawling works.

    Your site (if you talk about money site) is probably blocking the crawlers successfully, if you have the code in robots.txt. What you need to understand is that ahrefs and majestic don't need to crawl your website in order to display your external backlinks. The external backlinks are found and displayed by crawling the pages where the backlinks actually are.

    This can be the situation in your case:

    You have a PBN with backlinks to your money site. Since your PBN doesn't block crawlers, they crawled your network and display the backlinks pointing to your money site.

    Your competitor has also a PBN with backlinks to his money site. Since his PBN is blocking the crawlers, they cannot crawl his network to see where those websites are linking to.

    You cannot control what ahrefs/majestic displays unless you can control the websites where your backlink is located. So let's say both you and your competitor have a backlink on yahoo answers. In this case majestic will display (at different periods of time) the yahoo answers backlinks for both of you...regardless of what blocking codes you guys have on your money site.

    Hope this answered your question.
     
    • Thanks Thanks x 1
  5. VinceC

    VinceC Elite Member

    Joined:
    Jun 8, 2008
    Messages:
    1,733
    Likes Received:
    754
    Is this the way you thank ppl for replying you even thru that not the answer you looking for? dickhead
     
    • Thanks Thanks x 1
  6. dragiia

    dragiia BANNED BANNED

    Joined:
    Apr 28, 2014
    Messages:
    125
    Likes Received:
    116
    I appreciate it, that makes sense. However my competitor has a few of the same backlinks, however even those links do not show up on the ahrefs etc. If this was true, wouldn't those links show up as well?
     
  7. dragiia

    dragiia BANNED BANNED

    Joined:
    Apr 28, 2014
    Messages:
    125
    Likes Received:
    116
    He was obviously being condescending on purpose and his reply didn't help at all. Do you think anyone who responds deserves gratitude? English isn't your first language, I hope. Ignorant