1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Block majestic, ahrefs, moz, etc

Discussion in 'Black Hat SEO' started by KM1, Jul 27, 2017.

  1. KM1

    KM1 Newbie

    Joined:
    Mar 24, 2017
    Messages:
    22
    Likes Received:
    5
    Gender:
    Male
    Anybody has a robots.txt file to block out these crawlers?

    Majestic
    Moz
    Ahrefs
    Other major crawlers..

    I can make one, but it takes time. If anyone has one already, it will be much appreciated.

    Thanks!

    /K
     
  2. MatthewGraham

    MatthewGraham Jr. VIP Jr. VIP

    Joined:
    Oct 6, 2015
    Messages:
    540
    Likes Received:
    291
    • Thanks Thanks x 1
  3. SearchEngineWays

    SearchEngineWays Jr. VIP Jr. VIP

    Joined:
    Dec 3, 2014
    Messages:
    281
    Likes Received:
    80
    Gender:
    Male
    Occupation:
    SEARCH ENGINE MARKETING
    Location:
    Search Engine Result Page
    Here you can Block:

    Robots.txt:
    Code:
    User-agent: Rogerbot
    User-agent: Exabot
    User-agent: MJ12bot
    User-agent: Dotbot
    User-agent: Gigabot
    User-agent: AhrefsBot
    User-agent: BlackWidow
    User-agent: Bot\ [EMAIL="[email protected]"]mailto:[email protected][/EMAIL]
    User-agent: ChinaClaw
    User-agent: Custo
    User-agent: DISCo
    User-agent: Download\ Demon
    User-agent: eCatch
    User-agent: EirGrabber
    User-agent: EmailSiphon
    User-agent: EmailWolf
    User-agent: Express\ WebPictures
    User-agent: ExtractorPro
    User-agent: EyeNetIE
    User-agent: FlashGet
    User-agent: GetRight
    User-agent: GetWeb!
    User-agent: Go!Zilla
    User-agent: Go-Ahead-Got-It
    User-agent: GrabNet
    User-agent: Grafula
    User-agent: HMView
    User-agent: HTTrack
    User-agent: Image\ Stripper
    User-agent: Image\ Sucker
    User-agent: Indy\ Library
    User-agent: InterGET
    User-agent: Internet\ Ninja
    User-agent: JetCar
    User-agent: JOC\ Web\ Spider
    User-agent: larbin
    User-agent: LeechFTP
    User-agent: Mass\ Downloader
    User-agent: MIDown\ tool
    User-agent: Mister\ PiX
    User-agent: Navroad
    User-agent: NearSite
    User-agent: NetAnts
    User-agent: NetSpider
    User-agent: Net\ Vampire
    User-agent: NetZIP
    User-agent: Octopus
    User-agent: Offline\ Explorer
    User-agent: Offline\ Navigator
    User-agent: PageGrabber
    User-agent: Papa\ Foto
    User-agent: pavuk
    User-agent: pcBrowser
    User-agent: RealDownload
    User-agent: ReGet
    User-agent: SiteSnagger
    User-agent: SmartDownload
    User-agent: SuperBot
    User-agent: SuperHTTP
    User-agent: Surfbot
    User-agent: tAkeOut
    User-agent: Teleport\ Pro
    User-agent: VoidEYE
    User-agent: Web\ Image\ Collector
    User-agent: Web\ Sucker
    User-agent: WebAuto
    User-agent: WebCopier
    User-agent: WebFetch
    User-agent: WebGo\ IS
    User-agent: WebLeacher
    User-agent: WebReaper
    User-agent: WebSauger
    User-agent: Website\ eXtractor
    User-agent: Website\ Quester
    User-agent: WebStripper
    User-agent: WebWhacker
    User-agent: WebZIP
    User-agent: Wget
    User-agent: Widow
    User-agent: WWWOFFLE
    User-agent: Xaldon\ WebSpider
    User-agent: Zeus
    Disallow: /
    .htaccess:
    Code:
    SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
    SetEnvIfNoCase User-Agent .*exabot.* bad_bot
    SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
    SetEnvIfNoCase User-Agent .*mozbot.* bad_bot
    SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
    SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
    SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
    SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
    <Limit GET POST HEAD>
    Order Allow,Deny
    Allow from all
    Deny from env=bad_bot
    </Limit>
    
     
    • Thanks Thanks x 1