Block majestic, ahrefs, moz, etc

Discussion in 'Black Hat SEO' started by KM1, Jul 27, 2017.

  1. KM1

    KM1 Newbie

    Joined:
    Mar 24, 2017
    Messages:
    41
    Likes Received:
    13
    Gender:
    Male
    Anybody has a robots.txt file to block out these crawlers?

    Majestic
    Moz
    Ahrefs
    Other major crawlers..

    I can make one, but it takes time. If anyone has one already, it will be much appreciated.

    Thanks!

    /K
     
  2. MatthewGraham

    MatthewGraham Jr. VIP Jr. VIP

    Joined:
    Oct 6, 2015
    Messages:
    1,652
    Likes Received:
    2,019
    Gender:
    Male
    Occupation:
    Rolling Face on Keyboard
    Location:
    United States of America
    Home Page:
    • Thanks Thanks x 1
  3. SearchEngineWays

    SearchEngineWays Jr. VIP Jr. VIP

    Joined:
    Dec 3, 2014
    Messages:
    1,155
    Likes Received:
    201
    Gender:
    Male
    Occupation:
    SEARCH ENGINE MARKETING
    Location:
    Search Engine Result Page
    Home Page:
    Here you can Block:

    Robots.txt:
    Code:
    User-agent: Rogerbot
    User-agent: Exabot
    User-agent: MJ12bot
    User-agent: Dotbot
    User-agent: Gigabot
    User-agent: AhrefsBot
    User-agent: BlackWidow
    User-agent: Bot\ [EMAIL="[email protected]"]mailto:[email protected][/EMAIL]
    User-agent: ChinaClaw
    User-agent: Custo
    User-agent: DISCo
    User-agent: Download\ Demon
    User-agent: eCatch
    User-agent: EirGrabber
    User-agent: EmailSiphon
    User-agent: EmailWolf
    User-agent: Express\ WebPictures
    User-agent: ExtractorPro
    User-agent: EyeNetIE
    User-agent: FlashGet
    User-agent: GetRight
    User-agent: GetWeb!
    User-agent: Go!Zilla
    User-agent: Go-Ahead-Got-It
    User-agent: GrabNet
    User-agent: Grafula
    User-agent: HMView
    User-agent: HTTrack
    User-agent: Image\ Stripper
    User-agent: Image\ Sucker
    User-agent: Indy\ Library
    User-agent: InterGET
    User-agent: Internet\ Ninja
    User-agent: JetCar
    User-agent: JOC\ Web\ Spider
    User-agent: larbin
    User-agent: LeechFTP
    User-agent: Mass\ Downloader
    User-agent: MIDown\ tool
    User-agent: Mister\ PiX
    User-agent: Navroad
    User-agent: NearSite
    User-agent: NetAnts
    User-agent: NetSpider
    User-agent: Net\ Vampire
    User-agent: NetZIP
    User-agent: Octopus
    User-agent: Offline\ Explorer
    User-agent: Offline\ Navigator
    User-agent: PageGrabber
    User-agent: Papa\ Foto
    User-agent: pavuk
    User-agent: pcBrowser
    User-agent: RealDownload
    User-agent: ReGet
    User-agent: SiteSnagger
    User-agent: SmartDownload
    User-agent: SuperBot
    User-agent: SuperHTTP
    User-agent: Surfbot
    User-agent: tAkeOut
    User-agent: Teleport\ Pro
    User-agent: VoidEYE
    User-agent: Web\ Image\ Collector
    User-agent: Web\ Sucker
    User-agent: WebAuto
    User-agent: WebCopier
    User-agent: WebFetch
    User-agent: WebGo\ IS
    User-agent: WebLeacher
    User-agent: WebReaper
    User-agent: WebSauger
    User-agent: Website\ eXtractor
    User-agent: Website\ Quester
    User-agent: WebStripper
    User-agent: WebWhacker
    User-agent: WebZIP
    User-agent: Wget
    User-agent: Widow
    User-agent: WWWOFFLE
    User-agent: Xaldon\ WebSpider
    User-agent: Zeus
    Disallow: /
    .htaccess:
    Code:
    SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
    SetEnvIfNoCase User-Agent .*exabot.* bad_bot
    SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
    SetEnvIfNoCase User-Agent .*mozbot.* bad_bot
    SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
    SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
    SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
    SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
    <Limit GET POST HEAD>
    Order Allow,Deny
    Allow from all
    Deny from env=bad_bot
    </Limit>
    
     
    • Thanks Thanks x 1