ranga
Senior Member
- Aug 9, 2009
- 800
- 628
I am asked this question many times on how to hide our PBN from seo crawlers like Moz, Ahrefs, Majestic Seo etc. I had tried various solutions in the past including spyder spanker and provide below the details of how I hide my network.
1. Spyder Spanker
This was the first thing I tried, its a paid plugin and I used the classic version i.e. where you cannot block IPs only the user agents. This plugin was totally ineffective in blocking any crawlers. Infact, totally ineffective is wrong, I would say it was partially ineffective i.e. sometimes it used to block and sometimes it used to allow the crawlers, so the PBN links used to sometime appear and then disappear in the seo crawler stats, which is even worse as it gives people spying on the links even more suspicion that the website is using PBN links.
I tried contacting the developer that spyder spanker is not blocking the crawlers and got a cold answer "maybe its the theme or the hosting" without even looking into the issue. I use multiple host and multiple themes, so after getting such a reply I know its going to be a waste of time to just keep persuing the matter and its better to find another way.
2. Wordfence
This is free wordpress plugin and has all the features of spyder spanker pro without paying. I use this for its security and cache features and to block ips trying to hack into my website, but this is a must have plugin for any wordpress site.
This plugin can block user agents, ips, ip ranges, country plus has more features. So if you want to go the plugin route to block crawlers and bad bots then I would prefer wordfence to spyder spanker plus its free. For me its best and free spyder spanker alternative out there.
3. Htaccess
This is the best way to block the crawlers for me, I am using this method and it does block crawlers effectively and 100% of time as compared to spyder spanker. You can find out more about using this method here
http://www.blackhatworld.com/blackh...d-how-stop-competitors-spying-your-links.html
You however need to access your raw access logs to find out about the bots that are trying to access your websites and add those in the htaccess file.
I don't use robots.txt file as its totally ineffective in blocking spiders, plus it leaves a footprint.
I hope this helps you in making your PBN invisible.
1. Spyder Spanker
This was the first thing I tried, its a paid plugin and I used the classic version i.e. where you cannot block IPs only the user agents. This plugin was totally ineffective in blocking any crawlers. Infact, totally ineffective is wrong, I would say it was partially ineffective i.e. sometimes it used to block and sometimes it used to allow the crawlers, so the PBN links used to sometime appear and then disappear in the seo crawler stats, which is even worse as it gives people spying on the links even more suspicion that the website is using PBN links.
I tried contacting the developer that spyder spanker is not blocking the crawlers and got a cold answer "maybe its the theme or the hosting" without even looking into the issue. I use multiple host and multiple themes, so after getting such a reply I know its going to be a waste of time to just keep persuing the matter and its better to find another way.
2. Wordfence
This is free wordpress plugin and has all the features of spyder spanker pro without paying. I use this for its security and cache features and to block ips trying to hack into my website, but this is a must have plugin for any wordpress site.
This plugin can block user agents, ips, ip ranges, country plus has more features. So if you want to go the plugin route to block crawlers and bad bots then I would prefer wordfence to spyder spanker plus its free. For me its best and free spyder spanker alternative out there.
3. Htaccess
This is the best way to block the crawlers for me, I am using this method and it does block crawlers effectively and 100% of time as compared to spyder spanker. You can find out more about using this method here
http://www.blackhatworld.com/blackh...d-how-stop-competitors-spying-your-links.html
You however need to access your raw access logs to find out about the bots that are trying to access your websites and add those in the htaccess file.
I don't use robots.txt file as its totally ineffective in blocking spiders, plus it leaves a footprint.
I hope this helps you in making your PBN invisible.