Anybody with the code to block Majestic, Ahrefs and all others from crawling a site

csmbh said:

Is there any reason to not simply block every bot and just allow ones you want through?

ie. something like (adding other SE's & services you use):

User-agent: *
Disallow: /


User-agent: Googlebot
Allow: /

Click to expand...

srabantipaul1983 said:

so if we block those Bot then how will be Backlink Crawler Determine the Domain Metrics where we Block them..?? Like Moz DA/PA, Majestic CF/TF..?? that will be depend on the Backlink point to the PBN or any Site...

Click to expand...

srabantipaul1983 said:

so if we block those Bot then how will be Backlink Crawler Determine the Domain Metrics where we Block them..?? Like Moz DA/PA, Majestic CF/TF..?? that will be depend on the Backlink point to the PBN or any Site...

Click to expand...

aivaras said:

No, the goal is to block backlinks which you don't want to appear on your money site link profile on all the Ahrefs, Majestic and other tools, that would keep you away from competitors noticing any bad links and reporting you to G.

Click to expand...

Smeems said:

Ahrefs/Majestic/OSE are all backlink checkers. They crawl the site in question and map the inbound links pointing to that site. They don't (currently) add to that data by adding in additional links that they know are outbound links on other sites they have crawled. Would be an interesting feature but AFAIK they don't do it yet.

As a result, the crawlers' results are limited to the site crawl for that domain. If the bots are blocked from the domain from the outset, they can't report links.

Click to expand...

seoforaliving said:

I think blocking your money site is enough to tell crawlers not to index or report any backlinks about the sites in question.

Click to expand...

Pankage said:

Sorry for bringing this thread up, but I wanna know - is it late to use this as I already posted links on some domains and they can be seen on Majestic/ahrefs and other?

Click to expand...

Smeems said:

Here you go:

Robots.txt:

Code:

User-agent: Rogerbot 
User-agent: Exabot 
User-agent: MJ12bot 
User-agent: Dotbot 
User-agent: Gigabot 
User-agent: AhrefsBot 
User-agent: BlackWidow 
User-agent: Bot\ [EMAIL="[email protected]"]mailto:[email protected][/EMAIL] 
User-agent: ChinaClaw 
User-agent: Custo 
User-agent: DISCo 
User-agent: Download\ Demon 
User-agent: eCatch 
User-agent: EirGrabber 
User-agent: EmailSiphon 
User-agent: EmailWolf 
User-agent: Express\ WebPictures 
User-agent: ExtractorPro 
User-agent: EyeNetIE 
User-agent: FlashGet 
User-agent: GetRight 
User-agent: GetWeb! 
User-agent: Go!Zilla 
User-agent: Go-Ahead-Got-It 
User-agent: GrabNet 
User-agent: Grafula 
User-agent: HMView 
User-agent: HTTrack 
User-agent: Image\ Stripper 
User-agent: Image\ Sucker 
User-agent: Indy\ Library
User-agent: InterGET 
User-agent: Internet\ Ninja 
User-agent: JetCar 
User-agent: JOC\ Web\ Spider 
User-agent: larbin 
User-agent: LeechFTP 
User-agent: Mass\ Downloader 
User-agent: MIDown\ tool 
User-agent: Mister\ PiX 
User-agent: Navroad 
User-agent: NearSite 
User-agent: NetAnts 
User-agent: NetSpider 
User-agent: Net\ Vampire 
User-agent: NetZIP 
User-agent: Octopus 
User-agent: Offline\ Explorer 
User-agent: Offline\ Navigator 
User-agent: PageGrabber 
User-agent: Papa\ Foto 
User-agent: pavuk 
User-agent: pcBrowser 
User-agent: RealDownload 
User-agent: ReGet 
User-agent: SiteSnagger 
User-agent: SmartDownload 
User-agent: SuperBot 
User-agent: SuperHTTP 
User-agent: Surfbot 
User-agent: tAkeOut 
User-agent: Teleport\ Pro 
User-agent: VoidEYE 
User-agent: Web\ Image\ Collector 
User-agent: Web\ Sucker 
User-agent: WebAuto 
User-agent: WebCopier 
User-agent: WebFetch 
User-agent: WebGo\ IS 
User-agent: WebLeacher 
User-agent: WebReaper 
User-agent: WebSauger 
User-agent: Website\ eXtractor 
User-agent: Website\ Quester 
User-agent: WebStripper 
User-agent: WebWhacker 
User-agent: WebZIP 
User-agent: Wget 
User-agent: Widow 
User-agent: WWWOFFLE 
User-agent: Xaldon\ WebSpider 
User-agent: Zeus
Disallow: /

.htaccess:

Code:

SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
SetEnvIfNoCase User-Agent .*exabot.* bad_bot
SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>

Click to expand...

alwaystoday said:

Hi all, I am just about to build a PBN. One of the first things I am now looking into after reading,post #13 http://www.blackhatworld.com/blackh...manage-rank-pbns-post7132576.html#post7132576 in this QnA on PBNs - is how to block sites from crawling the PBN like ahrefs majestic etc.

I noticed the original code here was provided 3 years ago. Could someone with relevent experience let me know if this code is still good, or provide a better version which might be updated for 2016.

Also could you advise if you think there are any other precautions we should take when building a PBN to try to limit the crawling / snooping of unwanted sites etc.

Thanks

Click to expand...