Blocking All Crawlers Except of Google Question

[Engange]

Hi
I built myself a pretty solid backlinks profile,and I don't like my competitors to look at it.
Is it safe to block everyone but Google? Wouldn't that alert Google about suspicious website to check?

 

[choppa2choppa]

I remember seeing how to do this on Leith's sticky thread under section 2.. http://www.blackhatworld.com/blackh...rategies-tips-advice-inside-15-000-words.html

 

[Doolboy]

what is a crawler?

 

[jimbobo2779]

Hi
I built myself a pretty solid backlinks profile,and I don't like my competitors to look at it.
Is it safe to block everyone but Google? Wouldn't that alert Google about suspicious website to check?

Despite being a footprint of someone that knows about the SEO game I don't think it would signify that there is anything suspect or dodgy about the website.

There are legitimate reasons why someone may want to prevent crawling of their website such as:

• To save bandwidth
• To protect their copyright

Their are other legitimate reasons I am probably missing but I don't think it would be an issue to block them. I would advise to also allow other search engines though.

 

[Furious Man]

google it buddy you can get all info about crawler

what is a crawler?

 

[Ambitious12]

No why you want to block others,good brands never block others.

 

[SEO Sicko]

Crawler is used to identify the websites by themselves and scan the sites by links following from one site to another.

what is a crawler?

 

[Nerevar]

There's a plugin on wordpress.org for blocking crawlers and it's already used by more than 10,000 blogs.

 

[DigitalCon]

Put this in .htaccess file of every website you don't want to be crawled by bots like ahrefs and majestic. This should protect other crawlers to peek into your content. However at times i have experienced those bad bots to still crawl my websites somehow. They must be cloaking their user agents i suppose

[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot[/FONT][/COLOR][COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*exabot.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*dotbot.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*gigabot.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*sitebot.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*semrushbot.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*ia_archiver.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*searchmetricsbot.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*seokicks-robot.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*sistrix.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*lipperhey spider.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*ncbot.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*backlinkcrawler.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*archive.org_bot.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*meanpathbot.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*pagesinventory.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*aboundexbot.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*spbot.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*linkdexbot.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*nutch.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*blexbot.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*ezooms.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*scoutjet.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*majestic-12.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*majestic-seo.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*dsearch.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*blekkobo.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*screaming frog seo spider/*.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*PHPCrawl.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*gocrawl.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*DigExt.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*DomainSONOCrawler.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*TweetmemeBot.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*OpenHoseBot/2.1.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*Kraken/0.1.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*-Java-.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*ubermetrics.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*best-seo.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*Synapse.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*Harvest.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*Harvester.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*harvester.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*harvest.* bad_bot[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]
[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]<Limit GET POST HEAD> [/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]
[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]Order Allow,Deny [/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]
[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]Allow from all [/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]
[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]Deny from env=bad_bot [/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]
[/FONT][/COLOR]
[COLOR=#000000][FONT=Calibri]</Limit>[/FONT][/COLOR]

 

[Rachmaninoff]

For moneysite, there's no need to block other crawlers but google. Unless you have PBN links on your bl profile, you should just block the crawlers on your PBN. Not the moneysite.

 

[accelerator_dd]

If your backlink profile is your PBN, you can hide those links by blocking the crawling bots via htaccess as others mentioned above.

If the links are not on domains you control/have access to, you dont have many options. You could set a 301 to a new domain, and block the bots there, but I wouldn't advise it unless you really need to hide those links.