1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How to block spiders from majesticseo, ahrefs, etc?

Discussion in 'Black Hat SEO' started by yensidtlaw, Apr 29, 2014.

  1. yensidtlaw

    yensidtlaw Power Member

    Joined:
    Feb 25, 2013
    Messages:
    721
    Likes Received:
    474
    Occupation:
    "I play computers."
    Location:
    USA Baby!
    Hey all.

    I wanted to know how I can stop my backlinks from showing up in the backlink checking services such as majesticseo, ahrefs, etc? I obviously don't want my competition seeing where I'm getting my backlinks from and reporting me.

    I searched for an answer on Google and BHW, but all I could find was crap threads with no answers like this one: http://www.blackhatworld.com/blackh...seo-opensiteexplorer-other-backlink-bots.html

    Any help would be appreciated.
     
  2. ORJay

    ORJay Regular Member

    Joined:
    Aug 4, 2013
    Messages:
    291
    Likes Received:
    116
    Location:
    Mumbai
    You can use htaccess file +robot.txt combined, but the easiest way is using plugins like spyderspanker(paid) or link privacy (free)

    go to linkprivacy.com to download the plugin
     
  3. invest207

    invest207 Jr. VIP Jr. VIP Premium Member

    Joined:
    Apr 20, 2008
    Messages:
    542
    Likes Received:
    70
    Location:
    Mars
    It is possible to block spiders from indexing the links thare are on your pages. If you have a page that has a backllink on ...cnn.com lets say there is no way to stop that from indexing.
     
  4. yensidtlaw

    yensidtlaw Power Member

    Joined:
    Feb 25, 2013
    Messages:
    721
    Likes Received:
    474
    Occupation:
    "I play computers."
    Location:
    USA Baby!
    Ah ok. So there is no way to stop my competitors from seeing my links and reporting them to G? Fantastic...
     
  5. miedy

    miedy Senior Member

    Joined:
    May 17, 2012
    Messages:
    1,009
    Likes Received:
    463
    use this...

    Code:
    # ROBOTS.TXT STOPS BAD BOTS FROM CRAWLING YOUR WEB PAGES
    # PLEASE UPLOAD THIS FILE TO THE SAME FOLDER OF index.html
    
    
    User-agent: aipbot
    Disallow: /
    
    
    User-agent: ia_archiver
    Disallow: /
    
    
    User-agent: AhrefsBot
    Disallow: /
    
    
    User-agent: Alexibot 
    Disallow: /
    
    
    User-agent: Aqua_Products 
    Disallow: /
    
    
    User-agent: asterias 
    Disallow: /
    
    
    User-agent: b2w/0.1 
    Disallow: /
    
    
    User-agent: BackDoorBot/1.0 
    Disallow: /
    
    
    User-agent: becomebot
    Disallow: /
    
    
    User-agent: BlowFish/1.0 
    Disallow: /
    
    
    User-agent: Bookmark search tool 
    Disallow: /
    
    
    User-agent: BotALot 
    Disallow: /
    
    
    User-agent: BotRightHere 
    Disallow: /
    
    
    User-agent: BuiltBotTough 
    Disallow: /
    
    
    User-agent: Bullseye/1.0 
    Disallow: /
    
    
    User-agent: BunnySlippers 
    Disallow: /
    
    
    User-agent: CheeseBot 
    Disallow: /
    
    
    User-agent: CherryPicker 
    Disallow: /
    
    
    User-agent: CherryPickerElite/1.0 
    Disallow: /
    
    
    User-agent: CherryPickerSE/1.0 
    Disallow: /
    
    
    User-agent: Copernic 
    Disallow: /
    
    
    User-agent: CopyRightCheck 
    Disallow: /
    
    
    User-agent: cosmos 
    Disallow: /
    
    
    User-agent: Crescent 
    Disallow: /
    
    
    User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 
    Disallow: /
    
    
    User-agent: DittoSpyder 
    Disallow: /
    
    
    User-agent: dotbot
    Disallow: /
    
    
    User-agent: EmailCollector 
    Disallow: /
    
    
    User-agent: EmailSiphon 
    Disallow: /
    
    
    User-agent: EmailWolf 
    Disallow: /
    
    
    User-agent: EroCrawler 
    Disallow: /
    
    
    User-agent: exabot
    Disallow: /
    
    
    User-agent: ExtractorPro 
    Disallow: /
    
    
    User-agent: FairAd Client 
    Disallow: /
    
    
    User-agent: Fasterfox
    Disallow: /
    
    
    User-agent: Flaming AttackBot 
    Disallow: /
    
    
    User-agent: Foobot 
    Disallow: /
    
    
    User-agent: gigabot
    Disallow: /
    
    
    User-agent: Gaisbot 
    Disallow: /
    
    
    User-agent: GetRight/4.2 
    Disallow: /
    
    
    User-agent: Harvest/1.5 
    Disallow: /
    
    
    User-agent: hloader 
    Disallow: /
    
    
    User-agent: httplib 
    Disallow: /
    
    
    User-agent: HTTrack 3.0 
    Disallow: /
    
    
    User-agent: humanlinks 
    Disallow: /
    
    
    User-agent: ia_archiver
    Disallow: /
    
    
    User-agent: IconSurf
    Disallow: /
    Disallow: /favicon.ico
    
    
    User-agent: InfoNaviRobot 
    Disallow: /
    
    
    User-agent: Iron33/1.0.2 
    Disallow: /
    
    
    User-agent: JennyBot 
    Disallow: /
    
    
    User-agent: Kenjin Spider 
    Disallow: /
    
    
    User-agent: Keyword Density/0.9 
    Disallow: /
    
    
    User-agent: larbin 
    Disallow: /
    
    
    User-agent: LexiBot 
    Disallow: /
    
    
    User-agent: libWeb/clsHTTP 
    Disallow: /
    
    
    User-agent: LinkextractorPro 
    Disallow: /
    
    
    User-agent: LinkScan/8.1a Unix 
    Disallow: /
    
    
    User-agent: LinkWalker 
    Disallow: /
    
    
    User-agent: LNSpiderguy 
    Disallow: /
    
    
    User-agent: lwp-trivial 
    Disallow: /
    
    
    User-agent: lwp-trivial/1.34 
    Disallow: /
    
    
    User-agent: Mata Hari 
    Disallow: /
    
    
    User-agent: Microsoft URL Control 
    Disallow: /
    
    
    User-agent: Microsoft URL Control - 5.01.4511 
    Disallow: /
    
    
    User-agent: Microsoft URL Control - 6.00.8169 
    Disallow: /
    
    
    User-agent: MJ12bot
    Disallow: /
    
    
    User-agent: MIIxpc 
    Disallow: /
    
    
    User-agent: MIIxpc/4.2 
    Disallow: /
    
    
    User-agent: Mister PiX 
    Disallow: /
    
    
    User-agent: moget 
    Disallow: /
    
    
    User-agent: moget/2.1 
    Disallow: /
    
    
    User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95) 
    Disallow: /
    
    
    User-agent: MSIECrawler 
    Disallow: /
    
    
    User-agent: NetAnts 
    Disallow: /
    
    
    User-agent: NICErsPRO 
    Disallow: /
    
    
    User-agent: Offline Explorer 
    Disallow: /
    
    
    User-agent: Openbot 
    Disallow: /
    
    
    User-agent: Openfind 
    Disallow: /
    
    
    User-agent: Openfind data gatherer 
    Disallow: /
    
    
    User-agent: Oracle Ultra Search 
    Disallow: /
    
    
    User-agent: PerMan 
    Disallow: /
    
    
    User-agent: ProPowerBot/2.14 
    Disallow: /
    
    
    User-agent: ProWebWalker 
    Disallow: /
    
    
    User-agent: psbot 
    Disallow: /
    
    
    User-agent: Python-urllib 
    Disallow: /
    
    
    User-agent: QueryN Metasearch 
    Disallow: /
    
    
    User-agent: Radiation Retriever 1.1 
    Disallow: /
    
    
    User-agent: RepoMonkey 
    Disallow: /
    
    
    User-agent: RepoMonkey Bait & Tackle/v1.01 
    Disallow: /
    
    
    User-agent: RMA 
    Disallow: /
    
    
    User-agent: rogerbot
    disallow: /
    
    
    User-agent: searchpreview 
    Disallow: /
    
    
    User-agent: SiteSnagger 
    Disallow: /
    
    
    User-agent: SpankBot 
    Disallow: /
    
    
    User-agent: spanner 
    Disallow: /
    
    
    User-agent: SurveyBot
    Disallow: /
    
    
    User-agent: suzuran 
    Disallow: /
    
    
    User-agent: Szukacz/1.4 
    Disallow: /
    
    
    User-agent: Teleport 
    Disallow: /
    
    
    User-agent: TeleportPro 
    Disallow: /
    
    
    User-agent: Telesoft 
    Disallow: /
    
    
    User-agent: The Intraformant 
    Disallow: /
    
    
    User-agent: TheNomad 
    Disallow: /
    
    
    User-agent: TightTwatBot 
    Disallow: /
    
    
    User-agent: toCrawl/UrlDispatcher 
    Disallow: /
    
    
    User-agent: True_Robot 
    Disallow: /
    
    
    User-agent: True_Robot/1.0 
    Disallow: /
    
    
    User-agent: turingos 
    Disallow: /
    
    
    User-agent: TurnitinBot 
    Disallow: /
    
    
    User-agent: TurnitinBot/1.5 
    Disallow: /
    
    
    User-agent: URL Control 
    Disallow: /
    
    
    User-agent: URL_Spider_Pro 
    Disallow: /
    
    
    User-agent: URLy Warning 
    Disallow: /
    
    
    User-agent: VCI 
    Disallow: /
    
    
    User-agent: VCI WebViewer VCI WebViewer Win32 
    Disallow: /
    
    
    User-agent: Web Image Collector 
    Disallow: /
    
    
    User-agent: WebAuto 
    Disallow: /
    
    
    User-agent: WebBandit 
    Disallow: /
    
    
    User-agent: WebBandit/3.50 
    Disallow: /
    
    
    User-agent: WebCapture 2.0 
    Disallow: /
    
    
    User-agent: WebCopier 
    Disallow: /
    
    
    User-agent: WebCopier v.2.2 
    Disallow: /
    
    
    User-agent: WebCopier v3.2a 
    Disallow: /
    
    
    User-agent: WebEnhancer 
    Disallow: /
    
    
    User-agent: WebSauger 
    Disallow: /
    
    
    User-agent: Website Quester 
    Disallow: /
    
    
    User-agent: Webster Pro 
    Disallow: /
    
    
    User-agent: WebStripper 
    Disallow: /
    
    
    User-agent: WebZip 
    Disallow: /
    
    
    User-agent: WebZip 
    Disallow: /
    
    
    User-agent: WebZip/4.0 
    Disallow: /
    
    
    User-agent: WebZIP/4.21 
    Disallow: /
    
    
    User-agent: WebZIP/5.0 
    Disallow: /
    
    
    User-agent: Wget 
    Disallow: /
    
    
    User-agent: wget 
    Disallow: /
    
    
    User-agent: Wget/1.5.3 
    Disallow: /
    
    
    User-agent: Wget/1.6 
    Disallow: /
    
    
    User-agent: WWW-Collector-E 
    Disallow: /
    
    
    User-agent: Xenu's 
    Disallow: /
    
    
    User-agent: Xenu's Link Sleuth 1.1c 
    Disallow: /
    
    
    User-agent: Zeus 
    Disallow: /
    
    
    User-agent: Zeus 32297 Webster Pro V2.9 Win32 
    Disallow: /
    
    
    User-agent: Zeus Link Scout 
    Disallow: /
    
    
    User-agent: *
    Disallow: /js
    Disallow: *.js
    
    thats a long list when you want only to block ahrefs bot use this.

    Code:
    User-agent: AhrefsBot
    Disallow: /
     
    • Thanks Thanks x 4
  6. gr8mate

    gr8mate Newbie

    Joined:
    Apr 24, 2010
    Messages:
    43
    Likes Received:
    4
    Home Page:
    Been trying to block bots for a while but doesnt seem to be working this is my htaccess can anyone confirm if this works ..


    #htaccess mod rewrite code
    Options +FollowSymLinks -MultiViews
    RewriteEngine On
    RewriteBase /


    RewriteCond %{HTTP_USER_AGENT} ^Rogerbot* [OR]
    RewriteCond %{HTTP_USER_AGENT} ^Exabot* [OR]
    RewriteCond %{HTTP_USER_AGENT} ^MJ12bot* [OR]
    RewriteCond %{HTTP_USER_AGENT} ^Dotbot* [OR]
    RewriteCond %{HTTP_USER_AGENT} ^Gigabot* [OR]
    RewriteCond %{HTTP_USER_AGENT} ^AhrefsBot* [OR]
    RewriteCond %{HTTP_USER_AGENT} ^BlackWidow* [OR]
    RewriteRule ^(.*)$ http://www.bing.com/ [L,R=301]
     
  7. prab1996

    prab1996 Elite Member

    Joined:
    Jan 8, 2013
    Messages:
    3,496
    Likes Received:
    2,028
    Occupation:
    your gf's <3 ♥♥♥♥
    Location:
    Prab1996.com
    Home Page:
    There is no way to stop aherfs from indexing inbound links. you can only block it from indexing outbound links.

    -=-
     
  8. gr8mate

    gr8mate Newbie

    Joined:
    Apr 24, 2010
    Messages:
    43
    Likes Received:
    4
    Home Page:
    i have it on a redirect and main site which im linking to....
     
  9. Paul1427

    Paul1427 Registered Member

    Joined:
    Mar 24, 2012
    Messages:
    85
    Likes Received:
    2
    Gender:
    Male
    good job. It's work very good for me !
     
  10. resistosai111

    resistosai111 Power Member

    Joined:
    Oct 31, 2010
    Messages:
    550
    Likes Received:
    116
    Occupation:
    SEO
    Location:
    Somewhere in Earth
    Home Page:
    spyder spanker can do the same job.
     
  11. SEOmate

    SEOmate Jr. VIP Jr. VIP Premium Member

    Joined:
    Apr 24, 2014
    Messages:
    185
    Likes Received:
    31
    As i understand it, if you block Ahrefs and other services from seeing your links, your links won't be indexed and taken into consideration - as you weren't placing them at all.
    Nevermind competitors: if your links are good and nice then they'll have nothing to report.
    So better concentrade on wise and good linkbuilding.
    That's what SEO is about, if eveything was as easy as to simply hide your blackhat then anybody could do it, even a hamster. We wouldn't have needed this forum either.
     
  12. DamageX

    DamageX Elite Member

    Joined:
    Sep 17, 2008
    Messages:
    2,692
    Likes Received:
    1,687
    Occupation:
    Unemployable
    Location:
    Former nomad
    Don't bother with robots.txt, they just ignore it.

    Doesn't work. They have more bots than those, which don't display their useragent as bots.

    Don't quit your dayjob.
     
    • Thanks Thanks x 4