1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Blocking All Crawlers Except of Google Question

Discussion in 'Black Hat SEO' started by Engange, May 12, 2016.

  1. Engange

    Engange Regular Member

    Joined:
    Feb 27, 2013
    Messages:
    403
    Likes Received:
    41
    Hi
    I built myself a pretty solid backlinks profile,and I don't like my competitors to look at it.
    Is it safe to block everyone but Google? Wouldn't that alert Google about suspicious website to check?
     
  2. choppa2choppa

    choppa2choppa Registered Member

    Joined:
    Feb 22, 2016
    Messages:
    89
    Likes Received:
    12
  3. Doolboy

    Doolboy Registered Member

    Joined:
    Mar 2, 2015
    Messages:
    59
    Likes Received:
    6
    what is a crawler?
     
  4. jimbobo2779

    jimbobo2779 Jr. VIP Jr. VIP

    Joined:
    Sep 17, 2008
    Messages:
    3,732
    Likes Received:
    2,659
    Occupation:
    Software Engineer
    Location:
    UK
    Home Page:
    Despite being a footprint of someone that knows about the SEO game I don't think it would signify that there is anything suspect or dodgy about the website.

    There are legitimate reasons why someone may want to prevent crawling of their website such as:
    • To save bandwidth
    • To protect their copyright

    Their are other legitimate reasons I am probably missing but I don't think it would be an issue to block them. I would advise to also allow other search engines though.
     
  5. Furious Man

    Furious Man Jr. VIP Jr. VIP

    Joined:
    Aug 4, 2015
    Messages:
    1,739
    Likes Received:
    276
    google it buddy you can get all info about crawler

     
  6. Ambitious12

    Ambitious12 Elite Member

    Joined:
    Jun 26, 2014
    Messages:
    3,096
    Likes Received:
    609
    Occupation:
    No Occupation
    Location:
    Among the Stars
    No why you want to block others,good brands never block others.
     
  7. SEO Sicko

    SEO Sicko Power Member

    Joined:
    Oct 24, 2014
    Messages:
    778
    Likes Received:
    132
    Occupation:
    Digital marketer
    Home Page:
    Crawler is used to identify the websites by themselves and scan the sites by links following from one site to another.
     
  8. Nerevar

    Nerevar Jr. VIP Jr. VIP

    Joined:
    Jun 30, 2010
    Messages:
    482
    Likes Received:
    180
  9. DigitalCon

    DigitalCon Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 27, 2014
    Messages:
    519
    Likes Received:
    88
    Gender:
    Male
    Occupation:
    Internet Research
    Location:
    Home
    Home Page:
    Put this in .htaccess file of every website you don't want to be crawled by bots like ahrefs and majestic. This should protect other crawlers to peek into your content. However at times i have experienced those bad bots to still crawl my websites somehow. They must be cloaking their user agents i suppose :eek:

    Code:
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot[/FONT][/COLOR][COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*exabot.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*dotbot.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*gigabot.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*sitebot.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*semrushbot.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*ia_archiver.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*searchmetricsbot.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*seokicks-robot.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*sistrix.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*lipperhey spider.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*ncbot.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*backlinkcrawler.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*archive.org_bot.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*meanpathbot.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*pagesinventory.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*aboundexbot.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*spbot.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*linkdexbot.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*nutch.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*blexbot.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*ezooms.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*scoutjet.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*majestic-12.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*majestic-seo.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*dsearch.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*blekkobo.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*screaming frog seo spider/*.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*PHPCrawl.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*gocrawl.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*DigExt.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*DomainSONOCrawler.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*TweetmemeBot.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*OpenHoseBot/2.1.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*Kraken/0.1.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*-Java-.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*ubermetrics.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*best-seo.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*Synapse.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*Harvest.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*Harvester.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*harvester.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]SetEnvIfNoCase User-Agent .*harvest.* bad_bot[/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]
    [/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]<Limit GET POST HEAD> [/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]
    [/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]Order Allow,Deny [/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]
    [/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]Allow from all [/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]
    [/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]Deny from env=bad_bot [/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]
    [/FONT][/COLOR]
    [COLOR=#000000][FONT=Calibri]</Limit>[/FONT][/COLOR]
     
  10. LatteGrande

    LatteGrande Jr. VIP Jr. VIP Premium Member

    Joined:
    Jan 19, 2011
    Messages:
    2,197
    Likes Received:
    613
    Location:
    404 Not Found
    For moneysite, there's no need to block other crawlers but google. Unless you have PBN links on your bl profile, you should just block the crawlers on your PBN. Not the moneysite.
     
  11. accelerator_dd

    accelerator_dd Jr. VIP Jr. VIP

    Joined:
    May 14, 2010
    Messages:
    2,448
    Likes Received:
    1,010
    Occupation:
    SEO
    Location:
    IM Wonderland
    If your backlink profile is your PBN, you can hide those links by blocking the crawling bots via htaccess as others mentioned above.

    If the links are not on domains you control/have access to, you dont have many options. You could set a 301 to a new domain, and block the bots there, but I wouldn't advise it unless you really need to hide those links.