1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Want to protect your PBN? Then Help us out!!

Discussion in 'Black Hat SEO' started by codeman1234, Jul 20, 2014.

  1. codeman1234

    codeman1234 Power Member

    Joined:
    Sep 13, 2011
    Messages:
    527
    Likes Received:
    35
    Hello everyone,

    As you seen on my previous threads, I am looking for the name of bots like spyglass, ahrefs, etc to block them through .htaccess to protect our PBN!

    After been over a week with this, I am taking this to the next step, I am making a huge list with all bots I can find from software like majesticseo, ahrefs, etc where competitors can find our backlinks and either steal them or report us to Skynet (Google).

    I am finishing list and so far I got over 160 bots, I am going to post in about 20 hours once I debug it and add all others I have found.

    I would like to ask to everyone on this forum that is concern as me of protecting our PBN to please check the list see if there missing any bots and provide name of unknown bots like for example Seo Spyglass one that seems impossible to find.

    Also I know there some plugins for WP that work nice but, I prefer .htaccess file because personally not all my websites are on WP and .htaccess file can be use in all types of CMS or PHP/ASP/Ruby/etc. sites.

    The objective of this thread and the work I am trying to do here with your help is for us to create the perfect .htaccess antibot file and if there new bots on the future then we can release new versions of file on the future so we are always a step ahead.

    If you agree to help please let me know because we all want to protect our PBN and the best way to do it is as a community, like linux distributions do it and that is why linux is so good because community projects are the best way to reach perfection.

    Thanks everyone for reading this and I hope that together we can do the best .htaccess file outhere.


    Regards,

    Codeman
     
    • Thanks Thanks x 1
    Last edited: Jul 20, 2014
  2. classybabe

    classybabe Regular Member

    Joined:
    Mar 20, 2014
    Messages:
    247
    Likes Received:
    57
    Found this code in one of the posts here, mj12bot is majestic's bot.

    SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
    SetEnvIfNoCase User-Agent .*exabot.* bad_bot
    SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
    SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
    SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
    SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
    SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
    <Limit GET POST HEAD>
    Order Allow,Deny
    Allow from all
    Deny from env=bad_bot
    </Limit>
     
    • Thanks Thanks x 1
  3. xgnux

    xgnux Regular Member

    Joined:
    Sep 26, 2008
    Messages:
    492
    Likes Received:
    149
    Occupation:
    STudent
    Location:
    Germany
    I have done this myself before, have fun
    Also included many german backlink checkers not that much known to many us people :p

    Here you go:
    Code:
    RewriteEngine On
    RewriteCond %{HTTP_USER_AGENT} ^.*360Spider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Abonti [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*adsacomponent [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*AhrefsBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Alexibot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Aqua_Products [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*asterias [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*b2w [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*b2w/0\.1 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*BackDoorBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*BacklinkCrawler [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*baidu [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Baiduspider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*BlackWidow [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*BlekkoBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*BLEXBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*BlowFish [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*blo\.gs [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*blo\\\.gs [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Bookmark\\\ search\\\ tool [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*bot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*BotALot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Bot\\\ mailto\:craftbot@yahoo\.com [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*BuiltBotTough [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*bullseye [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*BunnySlippers [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*CareerBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*CheeseBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*CherryPicker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*cherry\ pick [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*cherry\\\ pick [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ChinaClaw [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*collect [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*CompSpyBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*copernic [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*CopyRightCheck [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*cosmos [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*crawl [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Crescent [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*cuil [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Custo [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*DCPbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*DISCo [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*DittoSpyder [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*domainsdb\.net [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*domainsdb\\\.net [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*dotbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Download\\\ Demon [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*dumbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*eCatch [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*EirGrabber [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*email [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*EmailCollector [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*EmailSiphon [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*emailwolf [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Enterprise_Search [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*EroCrawler [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*exabot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Express\\\ WebPictures [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ExtractorPro [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*EyeNetIE [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Ezooms [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*fairad\ client [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*fairad\\\ client [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Flaming\\\ AttackBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*FlashGet [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Foobot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*freefind [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Gaisbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*getright [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*GetWeb\! [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*gigabot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Go\!Zilla [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Go\-Ahead\-Got\-It [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*GrabNet [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Grafula [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*grub [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Harvest [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*hatena\ antenna [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Hatena\\\ Antenna [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*hloader [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*HMView [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*httplib [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*HTTrack [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*humanlinks [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ia_archiver [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Image\\\ Stripper [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Image\\\ Sucker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Indy\\\ Library [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Infohelfer [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*InfoNaviRobot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*InterGET [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Internet\\\ Ninja [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Iron33 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*jakarta [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*JennyBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Jetbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*JetCar [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*JOC\\\ Web\\\ Spider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Kenjin [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Keyword [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*keyword\ density [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*keyword\\\ density [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*larbin [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*LeechFTP [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*LexiBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*libweb [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*libWeb/clsHTTP [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*linkextractorpro [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*LinkScan [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*LinksManager\.com_bot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*LinksManager\\\.com_bot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*LinkWalker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*LNSpiderguy [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*looksmart [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*lwp\-trivial [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*lwp\\\-trivial [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*magpie\-crawler [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*magpie\\\-crawler [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*majestic\-12 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*majestic\\\-12 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Marketing [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Mass\\\ Downloader [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*mata\ hari [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Mata\\\ Hari [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*meanpathbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Microsoft\\\ URL\\\ Control [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*MIDown\\\ tool [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*miixpc [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*mister\ pix [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Mister\\\ PiX [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*MJ-12 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*MJ12 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*MJ12bot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*moget [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Moreover [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*MSIECrawler [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*naver [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Navroad [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*NearSite [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*NetAnts [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*NetMechanic [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*NetSpider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*NetZIP [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Net\\\ Vampire [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*nicerspro [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Nutch [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Octopus [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*offline\ explorer [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*offline\\\ explorer [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Offline\\\ Navigator [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ole\ control [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ole\\\ control [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Openbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*openfind [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*openfind\ data\ gathere [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*openfind\\\ data\\\ gathere [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*OpenindexSpider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Oracle\\\ Ultra\\\ Search [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Owlinbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*PageGrabber [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Papa\\\ Foto [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*pavuk [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*pcBrowser [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*perman [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ProCogSEOBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ProPowerBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ProWebWalker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*proximic [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*psbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*pycurl [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Python\-urllib [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*python\\\-urllib [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*QueryN [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*queryn\ metasearch [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*queryn\\\ metasearch [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*radiation\ retriever [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*radiation\\\ retriever [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Radiation\\\ Retriever\\\ 1\.1 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*RealDownload [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ReGet [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*repomonkey [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*RMA [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ROGER [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*rogerbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*scan [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*scooter [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Screaming [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*search [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SearchEngineWorld [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SearchmetricsBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*searchpreview [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Semrush [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SemrushBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SEO [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*seolytics [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*siphon [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*sistrix [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SiteExplorer [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*sitesnagger [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SmartDownload [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*sogou [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*sootle [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*soso [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SpankBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*spanner [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*spider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Spiderlytics [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*spy [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*spyder [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Squider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ssearch_bot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*stanford [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SuperBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SuperHTTP [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Surfbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SurveyBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*suzuran [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*szukacz [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*tAkeOut [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Teleport [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*teleportpro [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Telesoft [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*TheNomad [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*the\ intraformant [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*the\\\ intraformant [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*tocrawl [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*toCrawl/UrlDispatcher [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*True_Robot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*turingos [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*TurnitinBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*UnisterBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Updownerbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*updown_tester [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*urly\ warning [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*urly\\\ warning [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*url\ control [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*url\\\ control [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*url_spider_pro [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*VCI [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*VoidEYE [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebAuto [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*webbandit [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*webcopier [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebEnhancer [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebFetch [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebGo\\\ IS [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebLeacher [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebmasterWorld [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebReaper [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebSauger [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*website [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*website\ quester [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Website\\\ eXtractor [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Website\\\ Quester [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Webster [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*webster\ pro [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*webster\\\ pro [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebStripper [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebVac [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebViewer [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebWhacker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebZIP [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*web\ image\ collector [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*web\\\ image\\\ collector [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Web\\\ Sucker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*wget [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*whowhere [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Widow [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WWWOFFLE [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*www\-collector\-e [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*www\\\-collector\\\-e [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Xaldon\\\ WebSpider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Xenu [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*XoviBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Yandex [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*zeus [NC]
    RewriteRule ^ - [L,R=404]
     
    • Thanks Thanks x 7
  4. codeman1234

    codeman1234 Power Member

    Joined:
    Sep 13, 2011
    Messages:
    527
    Likes Received:
    35
    Hello Guys,

    Sorry for delay but, it took me a lot more than expected here is the list, I have found over 400 bots and still doing more also some of them are repeated because I had read that .htaccess is case sensitive, so we cover that too. Can someone please check list and let me know if there is anything else missing and syntax is all correct?


    Also I want to inform you that I found out Spyglass bot they use BLEXBot also added on list


    One question what is better to do with the bot redirect him to a 404 page or to a Forbidden Error?


    Let me know what you think version 0.1 of BHW .htaccess is been release :)


    Code:
    RewriteEngine On
    
    
    RewriteCond %{HTTP_USER_AGENT} ^.*360Spider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Abonti [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*adsacomponent [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*AhrefsBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ahrefsbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Alexibot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Aqua_Products [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*asterias [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*b2w [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*b2w/0\.1 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*b2w/0.1. [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*BackDoorBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*BacklinkCrawler [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*baidu [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Baiduspider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*BlackWidow [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*BlekkoBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*BLEXBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*BlowFish [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*blo\.gs [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*blo\\\.gs [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Bolt 0 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Bolt\ 0 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Bookmark\\\ search\\\ tool [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Bookmark\ search\ tool [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*bot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*BotALot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Bot\\\ mailto\:craftbot@yahoo\.com [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Bot\ mailto:craftbot@yahoo.com [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Bot mailto:craftbot@yahoo.com [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*BuiltBotTough [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*bullseye [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Bullseye [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*BunnySlippers [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*CareerBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*CazoodleBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*CheeseBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*CherryPicker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*cherry\ pick [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*cherry\\\ pick [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ChinaClaw [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*collect [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*CompSpyBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*copernic [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*CopyRightCheck [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*cosmos [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*crawl [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Crescent [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*cuil [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Custo [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*DCPbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Default Browser 0 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*DISCo [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*discobot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*DIIbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*DittoSpyder [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*domainsdb\.net [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*domainsdb\\\.net [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*dotbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Download Demon [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Download\ Demon [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Download\\\ Demon [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*dumbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*eCatch [NC,OR] 
    RewriteCond %{HTTP_USER_AGENT} ^.*EirGrabber [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*email [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*EmailCollector [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*EmailSiphon [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*emailwolf [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*EmailWolf [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Enterprise_Search [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*EroCrawler [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*exabot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Express WebPictures [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Express\\\ WebPictures [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Express\ WebPictures [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ExtractorPro [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ecxi [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*EyeNetIE [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Ezooms [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*FairAd [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*FairAd\\\ client [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*FairAd\ client [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*fairad\ client [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*fairad\\\ client [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Flaming\\\ AttackBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Flaming\ AttackBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*FlashGet [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Foobot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*freefind [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Gaisbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*getright [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*GetRight [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*GetWeb\! [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*GetWeb! [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*gigabot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Go\!Zilla [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Go!Zilla [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Go\-Ahead\-Got\-It [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Go-Ahead-Got-It [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*GrabNet [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Grafula [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*grub [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*GT::WWW [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Harvest [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*hatena\ antenna [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Hatena\\\ Antenna [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*heritrix [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*hloader [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*HMView [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*httplib [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*HTTrack [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*HTTP::Lite [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*humanlinks [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ia_archiver [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*IDBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*id-search [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*id-search.org [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Image Stripper [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Image\ Stripper [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Image\\\ Stripper [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Image\\\ Sucker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Image\ Sucker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Image Sucker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Indy Library [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Indy\ Library [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Indy\\\ Library [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Infohelfer [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*InfoNaviRobot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*InterGET [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Internet Ninja [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Internet\\\ Ninja [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Internet\ Ninja [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*InternetSeer.com [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*IRLbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Iron33 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ISC Systems iRc Search 2.1 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ISC\ Systems\ iRc\ Search\ 2.1 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*jakarta [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Java [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*JennyBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Jetbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*JetCar [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*JOC\\\ Web\\\ Spider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*JOC Web Spider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*JOC\ Web\ Spider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Kenjin [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Keyword [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*keyword\ density [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*keyword\\\ density [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*libwww-perl [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*libwww [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Link [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*LinksManager.com_bot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*larbin [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*LeechFTP [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*LexiBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*libweb [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*libWeb/clsHTTP [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*LinkextractorPro [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*linkextractorpro [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*LinkScan [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*LinksManager\.com_bot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*LinksManager\\\.com_bot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*LinkWalker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*linkwalker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*LNSpiderguy [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*looksmart [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*lwp-trivial [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*lwp\-trivial [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*lwp\\\-trivial [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*magpie\-crawler [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*magpie\\\-crawler [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*majestic\-12 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*majestic\\\-12 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Marketing [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Mass Downloader [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Mass\\\ Downloader [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Mass\ Downloader [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Mata\ Hari [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*mata\ hari [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Mata\\\ Hari [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Maxthon [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*meanpathbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*MFC_Tear_Sample [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Microsoft.URL [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Microsoft.Url [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*microsoft.url [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Microsoft URL Control [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Microsoft\ URL\ Control [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Microsoft\\\ URL\\\ Control [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*MIDown\\\ tool [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*MIDown\ tool [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*MIDown tool [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*MIIxpc [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*miixpc [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Mister PiX [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*mister\ pix [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Mister\ PiX [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Mister\\\ PiX [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Missigua Locator [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Missigua\ Locator [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*MJ-12 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*MJ12 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*MJ12bot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*mj12bot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*MJ12Bot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*moget [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Moreover [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Mozilla.*Indy [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Mozilla.*NEWT [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*MSIECrawler [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*MSFrontPage [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*naver [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Navroad [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*NearSite [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*NetAnts [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*NetMechanic [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*NetSpider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*NetZIP [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Net\\\ Vampire [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Net\ Vampire [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Net Vampire [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*nicerspro [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*NICErsPRO [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Nutch [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Octopus [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Offline\ Explorer [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Offline Explorer [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*offline\ explorer [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*offline\\\ explorer [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Offline\\\ Navigator [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Offline Navigator [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Offline\ Navigator [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ole\ control [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ole\\\ control [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Openbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Openfind [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*openfind [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*openfind\ data\ gathere [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*openfind\\\ data\\\ gathere [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*OpenindexSpider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Oracle\\\ Ultra\\\ Search [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Oracle\ Ultra\ Search [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Owlinbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*panscient.com [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*PageGrabber [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Papa\\\ Foto [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Papa\ Foto [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Papa Foto [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*pavuk [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*PECL::HTTP [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*PleaseCrawl [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*PeoplePal [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*pcBrowser [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*PHPCrawl [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*perman [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*PerMan [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ProCogSEOBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ProPowerBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ProWebWalker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*proximic [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*psbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*pycurl [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Python\-urllib [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*python\\\-urllib [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Python-urllib [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*QueryN [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*queryn\ metasearch [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*queryn\\\ metasearch [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*radiation\ retriever [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*radiation\\\ retriever [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Radiation\\\ Retriever\\\ 1\.1 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Radiation\ Retriever\ 1.1 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*RealDownload [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ReGet [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Rippers 0 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Rippers\ 0 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*repomonkey [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*RMA [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ROGER [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*rogerbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SBIder [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*scan [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*scooter [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Screaming [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SeaMonkey [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SeaMonkey$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*search [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SearchEngineWorld [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SearchmetricsBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*searchpreview [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Semrush [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SemrushBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SEO [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SEOkicks-Robot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*seolytics [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*siphon [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*sistrix [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*sitebot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Sitebot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SiteBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*sitecheck.internetseer.com [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SiteExplorer [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*sitesnagger [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SiteSnagger [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SmartDownload [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Snoopy [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Steeler [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*sogou [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*sootle [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*soso [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SpankBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*spanner [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*spider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Spiderlytics [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*spy [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*spyder [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Squider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ssearch_bot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*stanford [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SuperBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SuperHTTP [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Surfbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SurveyBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*suzuran [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*szukacz [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Szukacz [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*tAkeOut [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Teleport [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Teleport Pro[NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Teleport\ Pro [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*teleportpro [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Telesoft [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*TheNomad [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*the\ intraformant [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*The\ Intraformant [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*the\\\ intraformant [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Toata\ dragostea\ mea\ pentru\ diavola [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Toata dragostea mea pentru diavola [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*tocrawl [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*toCrawl/UrlDispatcher [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*True_Robot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*turingos [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*TurnitinBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*UnisterBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Updownerbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*updown_tester [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*URI::Fetch [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*urllib [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*urly\ warning [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*URLy\ Warning [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*urly\\\ warning [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*url\ control [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*url\\\ control [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*url_spider_pro [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*URL_Spider_Pro [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*User-Agent [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*VCI [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*VoidEYE [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*webalta [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebAuto [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*webbandit [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebBandit [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*webcopier [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebCopier [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebCollage [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebEnhancer [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebFetch [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebGo\\\ IS [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebGo\ IS [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebGo IS [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebLeacher [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebmasterWorld [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebmasterWorldForumBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebReaper [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebSauger [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*website [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*website\ quester [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Website\ Quester [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Website\\\ eXtractor [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Website\ eXtractor [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Website eXtractor [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Website\\\ Quester [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Website\ Quester [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Website Quester [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Webster [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*webster\ pro [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*webster\\\ pro [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebStripper [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebVac [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebViewer [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebWhacker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebZIP [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Wells Search II [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Wells\ Search\ II [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Web Image Collector [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Web\ Image\ Collector [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*web\ image\ collector [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*web\\\ image\\\ collector [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Web\\\ Sucker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Web Sucker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Web\ Sucker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*wget [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Wget [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*whowhere [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Widow [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WWWOFFLE [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WWW-Mechanize [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WWW-Collector-E [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*www\-collector\-e [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*www\\\-collector\\\-e [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*[Ww]eb[Bb]andit [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Xaldon\\\ WebSpider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Xaldon\ WebSpider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Xaldon WebSpider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Xenu [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*XoviBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Yandex [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*zermelo [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ZyBorg [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Zeus.*Webster [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Zeus [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*zeus [NC]
    RewriteRule ^ - [L,R=404]
    
    
    


    Also does anyone know the bots that this softwares use:

    KeywordSpy
    WordStream Keyword Tool
    Planificador de Palabras Clave de Google
    Ubersuggest Wordtracker
    Keyword Eye
    Keyword Discovery
    SEO Book Keyword Tool
    Advanced Web Ranking
    Market Samurai

    Since they are software also to analize competition


    Thanks,

    Codeman
     
    • Thanks Thanks x 3
    Last edited: Jul 22, 2014
  5. Known

    Known Regular Member

    Joined:
    Jan 27, 2013
    Messages:
    266
    Likes Received:
    187
    Occupation:
    IM
    Location:
    OH CANADA!!!!
    You could block their IP's too if you really wanted to be safe.
     
  6. codeman1234

    codeman1234 Power Member

    Joined:
    Sep 13, 2011
    Messages:
    527
    Likes Received:
    35
    Hello Known,

    Yes I was thinking about this too, but how I can find out their ips? Any ideas guys?

    Thanks for helping!
     
  7. GoTRooT

    GoTRooT Jr. VIP Jr. VIP

    Joined:
    Jun 21, 2010
    Messages:
    511
    Likes Received:
    241
    Occupation:
    Englland
    Location:
    Englland
    I am also very interested in this, currently building out my network and this is essential!

    (this thread will probably be ctrl+c / ctrl+v'd into a wso lol)
     
    • Thanks Thanks x 1
  8. wizdorf

    wizdorf Regular Member

    Joined:
    Jan 9, 2014
    Messages:
    221
    Likes Received:
    38
    Great list! Just to make sure these aren't blocking Google's bots right?
     
    • Thanks Thanks x 1
  9. ThopHayt

    ThopHayt Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 25, 2011
    Messages:
    5,396
    Likes Received:
    1,644
    Why not just ban all bots that aren't google/bing? Also is this not a footprint too lol?
     
  10. Velenterprise

    Velenterprise Registered Member

    Joined:
    Oct 5, 2012
    Messages:
    83
    Likes Received:
    19
    I am also very interested in this. Do keep us posted.
     
  11. cwvps

    cwvps Junior Member

    Joined:
    Dec 13, 2011
    Messages:
    139
    Likes Received:
    27
    Thanks for posting this, very cool and helpful.

     
  12. prab1996

    prab1996 Elite Member

    Joined:
    Jan 8, 2013
    Messages:
    3,496
    Likes Received:
    2,028
    Occupation:
    your gf's <3 ♥♥♥♥
    Location:
    Prab1996.com
    Home Page:
    thanks for sharing , but you not need to block any thing to protect your pbn, you just need to be clever and come up with new ideas.
    -=-
     
  13. TheUnborn

    TheUnborn Elite Member

    Joined:
    Feb 21, 2013
    Messages:
    3,041
    Likes Received:
    1,672
    Occupation:
    SEO Consultant
    Home Page:
    Excellent list,you are doing a great job.
     
  14. codeman1234

    codeman1234 Power Member

    Joined:
    Sep 13, 2011
    Messages:
    527
    Likes Received:
    35
    Dont worry google bot's is named googlebot, for bing is named bingbot, for yahoo is named yahoo!

    I am trying to block all bots that we don't win anything with their visit. Not a footprint just blocking the undesired :)

    Thanks, I will


    Thanks!

    We are all ears man!!! Share your clever ideas!


    -----------------------------------------------------------------------------------

    Ok guys I have found IPs for all bots so we can block them by IP too but I need some help and got some questions, if anyone can help I appreciatte:

    1) Do we block Yandex and Baidu search engine bots and other search engines or we unblock those?

    2) Is there other bots that we should not block?

    3) Anyone has any idea of a cool name we can call project like "BHW Majestic Bot Blocking List" :) any ideas?

    4) Do I send bots to 404 page or forbidden page?

    Please someone answer my questions so I can launch new file the fastest possible


    Also I am adding on file all IPs on file too, just give me some time I will try to post it today but looks more tomorrow since list is huge and I am trying to look for every software that any competitor not just backlinks profile software can use from email harvest, website copying, image copying, etc.

    Thanks,

    Codeman
     
    • Thanks Thanks x 1
    Last edited: Jul 23, 2014
  15. M4XW3LL

    M4XW3LL Senior Member

    Joined:
    Feb 5, 2013
    Messages:
    1,052
    Likes Received:
    1,247
    Awesome share man!
     
  16. codeman1234

    codeman1234 Power Member

    Joined:
    Sep 13, 2011
    Messages:
    527
    Likes Received:
    35
    Can someone please help me out answering my questions since I have found IP's for bots and is huge just and need some time to put it on file so please help me out with the answers:


    Thanks,

    Codeman