1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

PBN .htaccess Question

Discussion in 'Black Hat SEO' started by twpro, Dec 22, 2016.

  1. twpro

    twpro Jr. VIP Jr. VIP

    Joined:
    Jul 30, 2016
    Messages:
    263
    Likes Received:
    58
    Gender:
    Male
    Trying to block SEO crawlers using my .htaccess file. I have no idea how the file itself works but this is what I've come up with, it look okay?
    Thanks!
    Code:
    # BEGIN WordPress
    SetEnvIfNoCase User-Agent .*ia_archiver.* bad_bot
    SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
    SetEnvIfNoCase User-Agent .*exabot.* bad_bot
    SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
    SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
    SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
    SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
    SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
    SetEnvIfNoCase User-Agent .*xenu.* bad_bot
    <Limit GET POST HEAD>
    Order Allow,Deny
    Allow from all
    Deny from env=bad_bot
    </Limit>
    <IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteBase /
    RewriteRule ^index\.php$ - [L]
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteRule . /index.php [L]
    </IfModule>
    
    # END WordPress
    
     
  2. longerestcat

    longerestcat Newbie

    Joined:
    Nov 24, 2016
    Messages:
    27
    Likes Received:
    3
    Does this completely block bot access? if so that would be a bad thing wouldn't it...
     
  3. twpro

    twpro Jr. VIP Jr. VIP

    Joined:
    Jul 30, 2016
    Messages:
    263
    Likes Received:
    58
    Gender:
    Male
    If I did it correctly this should block most I can think of except google, it's favorable to prevent snoopers from finding all my PBN networks.
     
  4. longerestcat

    longerestcat Newbie

    Joined:
    Nov 24, 2016
    Messages:
    27
    Likes Received:
    3
    Oh I understand now... very nice...
     
  5. twpro

    twpro Jr. VIP Jr. VIP

    Joined:
    Jul 30, 2016
    Messages:
    263
    Likes Received:
    58
    Gender:
    Male
    Anyone?
     
  6. MikeyMikey13

    MikeyMikey13 Senior Member

    Joined:
    May 25, 2014
    Messages:
    1,047
    Likes Received:
    265
    Make sure your server has mod rewrite enabled. So many crappy hosts don't so this code won't even work.
     
    • Thanks Thanks x 1
  7. hardcorebiker

    hardcorebiker Jr. VIP Jr. VIP

    Joined:
    Sep 6, 2011
    Messages:
    1,335
    Likes Received:
    1,069
    Occupation:
    Digital Marketer
    Location:
    Bangkok, Thailand
    • Thanks Thanks x 2
  8. Nerevar

    Nerevar Jr. VIP Jr. VIP

    Joined:
    Jun 30, 2010
    Messages:
    475
    Likes Received:
    178
    • Thanks Thanks x 1
  9. agent-x

    agent-x Junior Member

    Joined:
    Apr 4, 2012
    Messages:
    173
    Likes Received:
    34
    Gender:
    Male
    The mod rewrite bit isn't for blocking bots, it's for Wordpress to handle database driven pages. So if it wasn't enabled, WP wouldn't work in the first place.
     
  10. twpro

    twpro Jr. VIP Jr. VIP

    Joined:
    Jul 30, 2016
    Messages:
    263
    Likes Received:
    58
    Gender:
    Male
  11. PFreely

    PFreely Newbie

    Joined:
    Dec 23, 2016
    Messages:
    15
    Likes Received:
    18
    Gender:
    Male
    Occupation:
    Lil bit of this, lil bit of that
    Location:
    US
    Yep - I've had this problem on multiple hosts. Otherwise, your .htaccess file looks fine, assuming your goal is to block bots.
     
    • Thanks Thanks x 2
  12. Dennie

    Dennie Jr. VIP Jr. VIP

    Joined:
    Nov 16, 2016
    Messages:
    122
    Likes Received:
    29
    Gender:
    Male
    There is another list here on the forum. Can't find the thread, but here is the list which was in the thread. It blocks a lot more bots then your list:

    Code:
    #block bots
    RewriteEngine On
    RewriteCond %{REQUEST_URI} !/robots.txt$
    RewriteCond %{HTTP_USER_AGENT} ^.*BLEXBot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*BlackWidow.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Nutch.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Jetbot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebVac.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Stanford.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*scooter.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*naver.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*dumbot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Hatena\ Antenna.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*grub.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*looksmart.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebZip.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*larbin.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*b2w/0.1.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Copernic.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*psbot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Python-urllib.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*NetMechanic.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*URL_Spider_Pro.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*CherryPicker.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*EmailCollector.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*EmailSiphon.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebBandit.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*EmailWolf.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Email.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ExtractorPro.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*CopyRightCheck.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Crescent.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SiteSnagger.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ProWebWalker.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*CheeseBot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*LNSpiderguy.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ia_archiver.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Alexibot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Teleport.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*MIIxpc.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Telesoft.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Website\ Quester.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*moget.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebStripper.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebSauger.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebCopier.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*NetAnts.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Mister\ PiX.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebAuto.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*TheNomad.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WWW-Collector-E.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*RMA.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*libWeb/clsHTTP.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*asterias.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*httplib.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*turingos.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*spanner.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Harvest.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*InfoNaviRobot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Bullseye.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebBandit.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*NICErsPRO.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Microsoft\ URL\ Control.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*DittoSpyder.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Foobot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebmasterWorldForumBot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SpankBot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*BotALot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*lwp-trivial.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebmasterWorld.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*BunnySlippers.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*URLy\ Warning.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Wget.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*LinkWalker.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*cosmos.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*hloader.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*humanlinks.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*LinkextractorPro.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Offline\ Explorer.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Mata\ Hari.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*LexiBot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Web\ Image\ Collector.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*The\ Intraformant.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*True_Robot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*BlowFish.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SearchEngineWorld.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*JennyBot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*MIIxpc.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*BuiltBotTough.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ProPowerBot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*BackDoorBot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*toCrawl/UrlDispatcher.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebEnhancer.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*suzuran.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebViewer.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*VCI.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Szukacz.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*QueryN.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Openfind.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Openbot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Webster.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*EroCrawler.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*LinkScan.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Keyword.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Kenjin.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Iron33.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Bookmark\ search\ tool.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*GetRight.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*FairAd\ Client.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Gaisbot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Aqua_Products.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Radiation\ Retriever\ 1.1.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Flaming\ AttackBot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Oracle\ Ultra\ Search.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*MSIECrawler.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*PerMan.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*searchpreview.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*sootle.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Enterprise_Search.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Bot\ mailto:[email protected]*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ChinaClaw.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Custo.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*DISCo.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Download\ Demon.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*eCatch.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*EirGrabber.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*EmailSiphon.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*EmailWolf.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Express\ WebPictures.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ExtractorPro.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*EyeNetIE.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*FlashGet.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*GetRight.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*GetWeb!.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Go!Zilla.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Go-Ahead-Got-It.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*GrabNet.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Grafula.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*HMView.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*HTTrack.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Image\ Stripper.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Image\ Sucker.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Indy\ Library.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*InterGET.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Internet\ Ninja.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*JetCar.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*JOC\ Web\ Spider.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*larbin.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*LeechFTP.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Mass\ Downloader.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*MIDown\ tool.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Mister\ PiX.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Navroad.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*NearSite.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*NetAnts.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*NetSpider.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Net\ Vampire.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*NetZIP.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Octopus.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Offline\ Explorer.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Offline\ Navigator.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*PageGrabber.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Papa\ Foto.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*pavuk.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*pcBrowser.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*RealDownload.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*ReGet.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SiteSnagger.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SmartDownload.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SuperBot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*SuperHTTP.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Surfbot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*tAkeOut.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Teleport\ Pro.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*VoidEYE.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Web\ Image\ Collector.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Web\ Sucker.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebAuto.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebCopier.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebFetch.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebGo\ IS.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebLeacher.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebReaper.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebSauger.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Website\ eXtractor.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Website\ Quester.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebStripper.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebWhacker.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WebZIP.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Wget.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Widow.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*WWWOFFLE.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Xaldon\ WebSpider.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Zeus.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Semrush.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*BecomeBot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*AhrefsBot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*MJ12bot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*rogerbot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*exabot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*Xenu.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*dotbot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*gigabot.*$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^.*BlekkoBot.*$ [NC]
     
    • Thanks Thanks x 2
  13. MikeyMikey13

    MikeyMikey13 Senior Member

    Joined:
    May 25, 2014
    Messages:
    1,047
    Likes Received:
    265
    You can have wordpress working fine, but they can disable modrewrite disabled for the whole server.

    It is the main reason PBN's still show up even with stuff like Link privacy or even this code.
     
    • Thanks Thanks x 1
  14. Steveputlocker

    Steveputlocker Junior Member

    Joined:
    Nov 3, 2016
    Messages:
    174
    Likes Received:
    18
    Gender:
    Male
    Bots are more likely to be blocked using robots.txt
     
  15. Dennie

    Dennie Jr. VIP Jr. VIP

    Joined:
    Nov 16, 2016
    Messages:
    122
    Likes Received:
    29
    Gender:
    Male
    Negative, most crawlers (bots) have the option to ignore the robots.txt. So robots is just to inform Google (and other search engines), but it is not guaranteed that it will respect what you have in the robots.

    So if you would like to block bots like majestic, semrush etc so others can't scan your pbn, you should definitely block them with htaccess.
     
    • Thanks Thanks x 1
  16. umerjutt00

    umerjutt00 Jr. VIP Jr. VIP

    Joined:
    Oct 28, 2011
    Messages:
    3,824
    Likes Received:
    2,063
    Occupation:
    Ninja
    Looks OK to me. Now just test it on your browser by changing your user-agent to some of those present in the htaccess.
     
    • Thanks Thanks x 1
  17. twpro

    twpro Jr. VIP Jr. VIP

    Joined:
    Jul 30, 2016
    Messages:
    263
    Likes Received:
    58
    Gender:
    Male
    @hardcorebiker @PFreely @Dennie @umerjutt00
    Thanks for the replies, working perfectly after testing with user-agent. Trying to do one more similar but different task and having issues:

    Trying to do two things.
    1. A complete domain 301 redirect using my .htaccess file
    2. Block googlebot from seeing the redirect(sounds dumb but I have a good reason)

    Here’s the code I have so far, it successfully blocks googlebot, but it is not redirecting.

    Code:
    RewriteEngine on
    RewriteCond %{HTTP_USER_AGENT} googlebot|yahoobot|microsoftbot [NC]
    RewriteRule ^.*$ – [R=403,L] Redirect 301 / http://www.MYNEWWEBSITEHERE.com/
    
    Any ideas what I’m doing wrong? The first rewritecond works, but the second line(the redirect) does not.
     
  18. Floopa75

    Floopa75 Jr. VIP Jr. VIP

    Joined:
    Feb 6, 2014
    Messages:
    814
    Likes Received:
    713
    Gender:
    Male
    OP everytime Wordpress updates it will rewrite that code without the blocking code in place. Meaning all your PBN links will show up in Ahrefs, Moz, etc. Always place blocking code after # END WordPress so this doesn't happen.
     
  19. twpro

    twpro Jr. VIP Jr. VIP

    Joined:
    Jul 30, 2016
    Messages:
    263
    Likes Received:
    58
    Gender:
    Male
    Thanks! Did not know that. Any idea about my code i posted in my last post?