1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How to allow ONLY normal users , google ,yahoo and bing only via .htaccess??

Discussion in 'Black Hat SEO' started by mirrorer, Aug 16, 2015.

  1. mirrorer

    mirrorer Jr. VIP Jr. VIP

    Joined:
    Jan 30, 2009
    Messages:
    1,311
    Likes Received:
    1,134
    Hello

    Does any know how to allow only normal users/visitors , google bot,yahoo & bing to crawl/visit a website? Is this possible at all?

    I did find this code

    Code:
    SetEnvIfNoCase User-Agent .*google.* search_robot
    SetEnvIfNoCase User-Agent .*yahoo.* search_robot
    SetEnvIfNoCase User-Agent .*bot.* search_robot
    SetEnvIfNoCase User-Agent .*ask.* search_robot
    
    
    Order Deny,Allow
    Deny from All
    Allow from env=search_robot
    
    But it blocks normal visitors and doesn't allow me to access my website.
     
  2. deancow

    deancow Jr. VIP Jr. VIP

    Joined:
    Jul 8, 2009
    Messages:
    682
    Likes Received:
    244
    you need to add your own useragent to that code i.e. firefox etc, theres so many of them it's better to just block all the bots you don't want to access the site rather than the other way round (theres plenty of examples in this forum)
     
  3. mirrorer

    mirrorer Jr. VIP Jr. VIP

    Joined:
    Jan 30, 2009
    Messages:
    1,311
    Likes Received:
    1,134
    I tried that but lot of crawlers are spoofing/faking their user agent
     
  4. myopic1

    myopic1 Regular Member

    Joined:
    Mar 24, 2014
    Messages:
    408
    Likes Received:
    404
    Of course you're blocking yourself, you've only allowed search engine user agents. Not only that but you may be blocking some other Google bots which don't broadcast themselves as Google.

    I suppose the main question here is, why? Your course of action is dependent on what it is you're trying to protect. If you're trying to prevent scraping, there are far more effective courses of action for you to take, if it's bandwidth there are rate limiting solutions....the salient point being that you need to provide more information before anyone can help you.