1. This website uses cookies to improve service and provide a tailored user experience. By using this site, you agree to this use. See our Cookie Policy.
    Dismiss Notice

Majestic indexed lots of PBN-Links recently. Wtf?!

Discussion in 'Black Hat SEO' started by kleinerwauwau, Jun 26, 2018.

  1. kleinerwauwau

    kleinerwauwau Junior Member

    Joined:
    Feb 7, 2017
    Messages:
    134
    Likes Received:
    13
    Gender:
    Male
    Hi guys,
    did you notice, that Majestic recently indexed lots of PBN-Links although the bot should be blocked via htaccess? Does anyone have information about that? It seems they've changed the name of their crawler or sth.
     
  2. Nut-Nights

    Nut-Nights Jr. VIP Jr. VIP

    Joined:
    Jun 20, 2013
    Messages:
    8,685
    Likes Received:
    5,334
    Occupation:
    Plumber
    Location:
    Hell
    Home Page:
    You thought they give a damn about htaccess?
     
    • Thanks Thanks x 1
  3. theRevolt

    theRevolt Jr. VIP Jr. VIP

    Joined:
    Jul 29, 2009
    Messages:
    2,146
    Likes Received:
    892
    Probably to do with moz changes , read here again
     
  4. kleinerwauwau

    kleinerwauwau Junior Member

    Joined:
    Feb 7, 2017
    Messages:
    134
    Likes Received:
    13
    Gender:
    Male
    It always worked for me. There was never one single indexed PBN-Site. I also know, that I'm by far not the only one who experiences sudden indexation by majestic atm.
     
  5. dhia27

    dhia27 Jr. VIP Jr. VIP

    Joined:
    Jan 26, 2016
    Messages:
    673
    Likes Received:
    318
    That's robots.txt you're talking about.

    As for htaccess, they are blocked by the server unless they use other user-agents.
     
  6. kleinerwauwau

    kleinerwauwau Junior Member

    Joined:
    Feb 7, 2017
    Messages:
    134
    Likes Received:
    13
    Gender:
    Male
    Why Moz? I'm talking about Majestic. BTW Ahrefs is still well-behaved and doesn't crawl any of my blocked PBN-Sites. What the hell did they change Majestic?
     
  7. theRevolt

    theRevolt Jr. VIP Jr. VIP

    Joined:
    Jul 29, 2009
    Messages:
    2,146
    Likes Received:
    892
    Oh I meant that majestic likely followed the Moz change to not be left behind by respecting robots.txt.

    While they do not mention htaccess in their post it would obviously be just another step to eliminate the "blind spots" they claim.
     
  8. kleinerwauwau

    kleinerwauwau Junior Member

    Joined:
    Feb 7, 2017
    Messages:
    134
    Likes Received:
    13
    Gender:
    Male
    Yeah, but they still shouldn't be able to access the website which blocked their crawler via htaccess unless they don't change their user agent. There's something fishy..
     
  9. agent-x

    agent-x Regular Member

    Joined:
    Apr 4, 2012
    Messages:
    253
    Likes Received:
    82
    Gender:
    Male
    Google and Bing do it, so why not Majestic?
     
  10. starki

    starki Senior Member

    Joined:
    Jul 17, 2012
    Messages:
    906
    Likes Received:
    340
    I think the blind spots are what many customers would like to see and since it's all about paying customers, they stepped up, probably crawling with various user agents. Since they use a bunch of totally different IPs anyway, blocking them will become impossible. Well, Google won't be unhappy about it.
     
  11. kleinerwauwau

    kleinerwauwau Junior Member

    Joined:
    Feb 7, 2017
    Messages:
    134
    Likes Received:
    13
    Gender:
    Male
    What a weak step from majestic. Ahrefs still has a lot better database without violating the rules I've made with the Htaccess.
     
  12. Peter Ngo

    Peter Ngo Jr. VIP Jr. VIP

    Joined:
    Apr 23, 2013
    Messages:
    2,758
    Likes Received:
    2,482
    Occupation:
    I browse BHW for a living
    Location:
    The Internet
    You sure you block it correctly? Most of my PBN are still hidden from what I checked.
     
  13. kleinerwauwau

    kleinerwauwau Junior Member

    Joined:
    Feb 7, 2017
    Messages:
    134
    Likes Received:
    13
    Gender:
    Male
    Of course. It worked perfectly ever since I've had Blog Network. Now suddenly majestic crawled almost all those links, also some of friends.
     
  14. starki

    starki Senior Member

    Joined:
    Jul 17, 2012
    Messages:
    906
    Likes Received:
    340
    Well, .htaccess rules can't be violated, just circumvented ;) Same result, but maybe Majestic does it to have some unique data others don't have, especially since Ahrefs is way ahead.
     
  15. kleinerwauwau

    kleinerwauwau Junior Member

    Joined:
    Feb 7, 2017
    Messages:
    134
    Likes Received:
    13
    Gender:
    Male
    The joke right there is, that Ahrefs database still is bigger without circumventing htaccess rules. This is Foul Play by Majestic.
     
  16. laur.laurix

    laur.laurix Senior Member

    Joined:
    May 8, 2013
    Messages:
    949
    Likes Received:
    385
    Occupation:
    Reverse Engineering Maniac
    Location:
    Mars
    The only way they can avoid .htaccess is changing their user agent or crawl from other ips like google and facebook.
     
  17. aidenhera

    aidenhera BANNED BANNED

    Joined:
    Nov 30, 2016
    Messages:
    3,060
    Likes Received:
    893
    Gender:
    Female
    i guess the safest choice is to block everything except Googlebots and browsers bots
     
  18. kleinerwauwau

    kleinerwauwau Junior Member

    Joined:
    Feb 7, 2017
    Messages:
    134
    Likes Received:
    13
    Gender:
    Male
    That's what I'm doing right now. Should've done that right from the start
     
  19. Buzzika

    Buzzika Supreme Member

    Joined:
    Jul 8, 2009
    Messages:
    1,425
    Likes Received:
    1,733
    Gender:
    Male
    Occupation:
    Hustler
    Location:
    Gurgaon
    Foul plaay?
    It's on the internet for god's sake. Sure ahrefs might be better at it.
    But you can't blame a link database for trying to expand it's data.
     
  20. redarrow

    redarrow Elite Member

    Joined:
    Apr 1, 2013
    Messages:
    10,753
    Likes Received:
    3,143
    show the htaccess your useing to do the blocking .