How to find backlinks that are blocking spiders?

darkshadow1 · Sep 16, 2014

Hi,

I have a 2 quick question. Will also take pm if it's a touchie subject.

Essentially I believe that a lot of my competitors are blocking ahrefs, majestic, etc...
From my understanding there are two ways of doing this.

1) in the robot.txt
2) .htaccess

I don't see anything in the robot.txt file that would indicate such, therefore I believe it is done via .htaccess

My questions are as follows.

1) How can I verify if they are blocking spiders?
I mean ranking for 1st or 2nd position without any deeplinks is a pretty good indicator, but still.... I would like to confirm that technically.

2) What approach to take to find some of those backlinks?

Thanks a lot in advance!

SEO Power · Sep 16, 2014

You didn't see anything in their robots.txt files because they do not place the code in the robots.txt files of their money sites. They do it in the robots.txt files of the sites providing the backlinks to their money sites, usually PBN sites. So, unfortunately, you can't technically verify that.

The only way to find out what links they are hiding is using different link research tools with the hope that they failed to block at least one link research tool. You can also use the link:domain.com operator to uncover some of their links since they can't hide their backlinks from Google.

ThopHayt · Sep 16, 2014

Robot blocking is voluntary, it just flags robots not to crawl, it does not prevent anything if a robot ignores. You could just send your own robots and ignore flags.

tony_d · Sep 17, 2014

If you want to find all the links, even those on sites blocking bots, you'll need to crawl the entire interwebs yourself, and make sure to disregard robots.txt.

darkshadow1 · Sep 17, 2014

SEO Power said:
You didn't see anything in their robots.txt files because they do not place the code in the robots.txt files of their money sites. They do it in the robots.txt files of the sites providing the backlinks to their money sites, usually PBN sites. So, unfortunately, you can't technically verify that.

Ok, I guess I misunderstood the source of where the links are being blocked, but it makes perfect sense now, thanks for the correction.

SEO Power said:
The only way to find out what links they are hiding is using different link research tools with the hope that they failed to block at least one link research tool. You can also use the link:domain.com operator to uncover some of their links since they can't hide their backlinks from Google.

yeah tried that but not having much success with that.

darkshadow1 · Sep 17, 2014

ThopHayt said:
Robot blocking is voluntary, it just flags robots not to crawl, it does not prevent anything if a robot ignores. You could just send your own robots and ignore flags.

Aha, ok will look into that. Sounds promising, Thanks!

darkshadow1 · Sep 17, 2014

tony_d said:
If you want to find all the links, even those on sites blocking bots, you'll need to crawl the entire interwebs yourself, and make sure to disregard robots.txt.

starting to get the picture, will but some real thought into it. Thanks!

darkshadow1 · Sep 17, 2014

Thanks for the replies. I have basic understanding of how to approach this. If any of you have gone down this route I would like to ask if you are running in your own environment or hosted environment?
Also, using any proxies for this?
Any issues with ISP due to all the traffic?

SEO Power · Sep 17, 2014

darkshadow1 said:
Thanks for the replies. I have basic understanding of how to approach this. If any of you have gone down this route I would like to ask if you are running in your own environment or hosted environment?
Also, using any proxies for this?
Any issues with ISP due to all the traffic?

The truth is that there's little you can do to uncover the backlink sources of sites blocking link crawlers. The only way that would really work is hacking the money site, adding it to GWT, and verifying it. Then you'll see all it's backlinks through GWT backlinks report. However, that's unethical.

boombaby7 · Sep 17, 2014

thats a lie. From no hat digital.

Sneaky way to uncover the PBN of someone spider blocking: search ?domain.com? -site:domain.com. This will help you to find any instances of a domain name being being used as the anchor text (quite a common way to link) on PBN sites. For example, to find mentions of our site, search this:

?nohatdigital.com? -site:nohatdigital.com

boombaby7 · Sep 17, 2014

obviously isn't perfect, but it uncovers some url anchor text links that majestic or ahrefs may have not picked up.

How to find backlinks that are blocking spiders?

darkshadow1

Junior Member

SEO Power

Elite Member

ThopHayt

Elite Member

tony_d

Elite Member

darkshadow1

Junior Member

darkshadow1

Junior Member

darkshadow1

Junior Member

darkshadow1

Junior Member

SEO Power

Elite Member

boombaby7

Junior Member

boombaby7

Junior Member

Main Menu

Marketplace

Making Money

BlackHat World