How to block spiders from majesticseo, ahrefs, etc?

yensidtlaw · Apr 29, 2014

Hey all.

I wanted to know how I can stop my backlinks from showing up in the backlink checking services such as majesticseo, ahrefs, etc? I obviously don't want my competition seeing where I'm getting my backlinks from and reporting me.

I searched for an answer on Google and BHW, but all I could find was crap threads with no answers like this one: http://www.blackhatworld.com/blackh...seo-opensiteexplorer-other-backlink-bots.html

Any help would be appreciated.

ORJay · Apr 29, 2014

You can use htaccess file +robot.txt combined, but the easiest way is using plugins like spyderspanker(paid) or link privacy (free)

go to linkprivacy.com to download the plugin

invest207 · Apr 29, 2014

It is possible to block spiders from indexing the links thare are on your pages. If you have a page that has a backllink on ...cnn.com lets say there is no way to stop that from indexing.

yensidtlaw · Apr 29, 2014

invest207 said:
It is possible to block spiders from indexing the links thare are on your pages. If you have a page that has a backllink on ...cnn.com lets say there is no way to stop that from indexing.

Ah ok. So there is no way to stop my competitors from seeing my links and reporting them to G? Fantastic...

miedy · Apr 29, 2014

use this...

Code:

# ROBOTS.TXT STOPS BAD BOTS FROM CRAWLING YOUR WEB PAGES
# PLEASE UPLOAD THIS FILE TO THE SAME FOLDER OF index.html


User-agent: aipbot
Disallow: /


User-agent: ia_archiver
Disallow: /


User-agent: AhrefsBot
Disallow: /


User-agent: Alexibot 
Disallow: /


User-agent: Aqua_Products 
Disallow: /


User-agent: asterias 
Disallow: /


User-agent: b2w/0.1 
Disallow: /


User-agent: BackDoorBot/1.0 
Disallow: /


User-agent: becomebot
Disallow: /


User-agent: BlowFish/1.0 
Disallow: /


User-agent: Bookmark search tool 
Disallow: /


User-agent: BotALot 
Disallow: /


User-agent: BotRightHere 
Disallow: /


User-agent: BuiltBotTough 
Disallow: /


User-agent: Bullseye/1.0 
Disallow: /


User-agent: BunnySlippers 
Disallow: /


User-agent: CheeseBot 
Disallow: /


User-agent: CherryPicker 
Disallow: /


User-agent: CherryPickerElite/1.0 
Disallow: /


User-agent: CherryPickerSE/1.0 
Disallow: /


User-agent: Copernic 
Disallow: /


User-agent: CopyRightCheck 
Disallow: /


User-agent: cosmos 
Disallow: /


User-agent: Crescent 
Disallow: /


User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 
Disallow: /


User-agent: DittoSpyder 
Disallow: /


User-agent: dotbot
Disallow: /


User-agent: EmailCollector 
Disallow: /


User-agent: EmailSiphon 
Disallow: /


User-agent: EmailWolf 
Disallow: /


User-agent: EroCrawler 
Disallow: /


User-agent: exabot
Disallow: /


User-agent: ExtractorPro 
Disallow: /


User-agent: FairAd Client 
Disallow: /


User-agent: Fasterfox
Disallow: /


User-agent: Flaming AttackBot 
Disallow: /


User-agent: Foobot 
Disallow: /


User-agent: gigabot
Disallow: /


User-agent: Gaisbot 
Disallow: /


User-agent: GetRight/4.2 
Disallow: /


User-agent: Harvest/1.5 
Disallow: /


User-agent: hloader 
Disallow: /


User-agent: httplib 
Disallow: /


User-agent: HTTrack 3.0 
Disallow: /


User-agent: humanlinks 
Disallow: /


User-agent: ia_archiver
Disallow: /


User-agent: IconSurf
Disallow: /
Disallow: /favicon.ico


User-agent: InfoNaviRobot 
Disallow: /


User-agent: Iron33/1.0.2 
Disallow: /


User-agent: JennyBot 
Disallow: /


User-agent: Kenjin Spider 
Disallow: /


User-agent: Keyword Density/0.9 
Disallow: /


User-agent: larbin 
Disallow: /


User-agent: LexiBot 
Disallow: /


User-agent: libWeb/clsHTTP 
Disallow: /


User-agent: LinkextractorPro 
Disallow: /


User-agent: LinkScan/8.1a Unix 
Disallow: /


User-agent: LinkWalker 
Disallow: /


User-agent: LNSpiderguy 
Disallow: /


User-agent: lwp-trivial 
Disallow: /


User-agent: lwp-trivial/1.34 
Disallow: /


User-agent: Mata Hari 
Disallow: /


User-agent: Microsoft URL Control 
Disallow: /


User-agent: Microsoft URL Control - 5.01.4511 
Disallow: /


User-agent: Microsoft URL Control - 6.00.8169 
Disallow: /


User-agent: MJ12bot
Disallow: /


User-agent: MIIxpc 
Disallow: /


User-agent: MIIxpc/4.2 
Disallow: /


User-agent: Mister PiX 
Disallow: /


User-agent: moget 
Disallow: /


User-agent: moget/2.1 
Disallow: /


User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95) 
Disallow: /


User-agent: MSIECrawler 
Disallow: /


User-agent: NetAnts 
Disallow: /


User-agent: NICErsPRO 
Disallow: /


User-agent: Offline Explorer 
Disallow: /


User-agent: Openbot 
Disallow: /


User-agent: Openfind 
Disallow: /


User-agent: Openfind data gatherer 
Disallow: /


User-agent: Oracle Ultra Search 
Disallow: /


User-agent: PerMan 
Disallow: /


User-agent: ProPowerBot/2.14 
Disallow: /


User-agent: ProWebWalker 
Disallow: /


User-agent: psbot 
Disallow: /


User-agent: Python-urllib 
Disallow: /


User-agent: QueryN Metasearch 
Disallow: /


User-agent: Radiation Retriever 1.1 
Disallow: /


User-agent: RepoMonkey 
Disallow: /


User-agent: RepoMonkey Bait & Tackle/v1.01 
Disallow: /


User-agent: RMA 
Disallow: /


User-agent: rogerbot
disallow: /


User-agent: searchpreview 
Disallow: /


User-agent: SiteSnagger 
Disallow: /


User-agent: SpankBot 
Disallow: /


User-agent: spanner 
Disallow: /


User-agent: SurveyBot
Disallow: /


User-agent: suzuran 
Disallow: /


User-agent: Szukacz/1.4 
Disallow: /


User-agent: Teleport 
Disallow: /


User-agent: TeleportPro 
Disallow: /


User-agent: Telesoft 
Disallow: /


User-agent: The Intraformant 
Disallow: /


User-agent: TheNomad 
Disallow: /


User-agent: TightTwatBot 
Disallow: /


User-agent: toCrawl/UrlDispatcher 
Disallow: /


User-agent: True_Robot 
Disallow: /


User-agent: True_Robot/1.0 
Disallow: /


User-agent: turingos 
Disallow: /


User-agent: TurnitinBot 
Disallow: /


User-agent: TurnitinBot/1.5 
Disallow: /


User-agent: URL Control 
Disallow: /


User-agent: URL_Spider_Pro 
Disallow: /


User-agent: URLy Warning 
Disallow: /


User-agent: VCI 
Disallow: /


User-agent: VCI WebViewer VCI WebViewer Win32 
Disallow: /


User-agent: Web Image Collector 
Disallow: /


User-agent: WebAuto 
Disallow: /


User-agent: WebBandit 
Disallow: /


User-agent: WebBandit/3.50 
Disallow: /


User-agent: WebCapture 2.0 
Disallow: /


User-agent: WebCopier 
Disallow: /


User-agent: WebCopier v.2.2 
Disallow: /


User-agent: WebCopier v3.2a 
Disallow: /


User-agent: WebEnhancer 
Disallow: /


User-agent: WebSauger 
Disallow: /


User-agent: Website Quester 
Disallow: /


User-agent: Webster Pro 
Disallow: /


User-agent: WebStripper 
Disallow: /


User-agent: WebZip 
Disallow: /


User-agent: WebZip 
Disallow: /


User-agent: WebZip/4.0 
Disallow: /


User-agent: WebZIP/4.21 
Disallow: /


User-agent: WebZIP/5.0 
Disallow: /


User-agent: Wget 
Disallow: /


User-agent: wget 
Disallow: /


User-agent: Wget/1.5.3 
Disallow: /


User-agent: Wget/1.6 
Disallow: /


User-agent: WWW-Collector-E 
Disallow: /


User-agent: Xenu's 
Disallow: /


User-agent: Xenu's Link Sleuth 1.1c 
Disallow: /


User-agent: Zeus 
Disallow: /


User-agent: Zeus 32297 Webster Pro V2.9 Win32 
Disallow: /


User-agent: Zeus Link Scout 
Disallow: /


User-agent: *
Disallow: /js
Disallow: *.js

thats a long list when you want only to block ahrefs bot use this.

Code:

User-agent: AhrefsBot
Disallow: /

gr8mate · May 6, 2014

Been trying to block bots for a while but doesnt seem to be working this is my htaccess can anyone confirm if this works ..

#htaccess mod rewrite code
Options +FollowSymLinks -MultiViews
RewriteEngine On
RewriteBase /

RewriteCond %{HTTP_USER_AGENT} ^Rogerbot* [OR]
RewriteCond %{HTTP_USER_AGENT} ^Exabot* [OR]
RewriteCond %{HTTP_USER_AGENT} ^MJ12bot* [OR]
RewriteCond %{HTTP_USER_AGENT} ^Dotbot* [OR]
RewriteCond %{HTTP_USER_AGENT} ^Gigabot* [OR]
RewriteCond %{HTTP_USER_AGENT} ^AhrefsBot* [OR]
RewriteCond %{HTTP_USER_AGENT} ^BlackWidow* [OR]
RewriteRule ^(.*)$ http://www.bing.com/ [L,R=301]

prab1996 · May 6, 2014

There is no way to stop aherfs from indexing inbound links. you can only block it from indexing outbound links.

-=-

gr8mate · May 6, 2014

i have it on a redirect and main site which im linking to....

Paul1427 · May 6, 2014

gr8mate said:
Been trying to block bots for a while but doesnt seem to be working this is my htaccess can anyone confirm if this works ..

#htaccess mod rewrite code
Options +FollowSymLinks -MultiViews
RewriteEngine On
RewriteBase /

RewriteCond %{HTTP_USER_AGENT} ^Rogerbot* [OR]
RewriteCond %{HTTP_USER_AGENT} ^Exabot* [OR]
RewriteCond %{HTTP_USER_AGENT} ^MJ12bot* [OR]
RewriteCond %{HTTP_USER_AGENT} ^Dotbot* [OR]
RewriteCond %{HTTP_USER_AGENT} ^Gigabot* [OR]
RewriteCond %{HTTP_USER_AGENT} ^AhrefsBot* [OR]
RewriteCond %{HTTP_USER_AGENT} ^BlackWidow* [OR]
RewriteRule ^(.*)$ / [L,R=301]

good job. It's work very good for me !

resistosai111 · May 6, 2014

spyder spanker can do the same job.

SEOmate · May 8, 2014

As i understand it, if you block Ahrefs and other services from seeing your links, your links won't be indexed and taken into consideration - as you weren't placing them at all.
Nevermind competitors: if your links are good and nice then they'll have nothing to report.
So better concentrade on wise and good linkbuilding.
That's what SEO is about, if eveything was as easy as to simply hide your blackhat then anybody could do it, even a hamster. We wouldn't have needed this forum either.

DamageX · May 8, 2014

miedy said:
use this...

Don't bother with robots.txt, they just ignore it.

gr8mate said:
Been trying to block bots for a while but doesnt seem to be working this is my htaccess can anyone confirm if this works ..

#htaccess mod rewrite code
Options +FollowSymLinks -MultiViews
RewriteEngine On
RewriteBase /

RewriteCond %{HTTP_USER_AGENT} ^Rogerbot* [OR]
RewriteCond %{HTTP_USER_AGENT} ^Exabot* [OR]
RewriteCond %{HTTP_USER_AGENT} ^MJ12bot* [OR]
RewriteCond %{HTTP_USER_AGENT} ^Dotbot* [OR]
RewriteCond %{HTTP_USER_AGENT} ^Gigabot* [OR]
RewriteCond %{HTTP_USER_AGENT} ^AhrefsBot* [OR]
RewriteCond %{HTTP_USER_AGENT} ^BlackWidow* [OR]
RewriteRule ^(.*)$ / [L,R=301]

Doesn't work. They have more bots than those, which don't display their useragent as bots.

SEOmate said:
As i understand it, if you block Ahrefs and other services from seeing your links, your links won't be indexed and taken into consideration - as you weren't placing them at all.
Nevermind competitors: if your links are good and nice then they'll have nothing to report.
So better concentrade on wise and good linkbuilding.
That's what SEO is about, if eveything was as easy as to simply hide your blackhat then anybody could do it, even a hamster. We wouldn't have needed this forum either.

Don't quit your dayjob.

How to block spiders from majesticseo, ahrefs, etc?

yensidtlaw

Power Member

ORJay

Regular Member

invest207

Senior Member

yensidtlaw

Power Member

miedy

Supreme Member

gr8mate

Newbie

prab1996

Elite Member

gr8mate

Newbie

Paul1427

Junior Member

resistosai111

Senior Member

SEOmate

Regular Member

DamageX

Elite Member

Main Menu

Marketplace

Making Money

BlackHatWorld

Other