1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Will this robots file block all search engines?

Discussion in 'Black Hat SEO' started by Holla1, Dec 18, 2014.

  1. Holla1

    Holla1 Regular Member

    Joined:
    Aug 26, 2014
    Messages:
    230
    Likes Received:
    4
    User-agent: *
    Disallow: /


    This is what it says on google.
     
  2. Tobbe co

    Tobbe co Junior Member

    Joined:
    Sep 29, 2014
    Messages:
    171
    Likes Received:
    139
    Yes .
     
  3. BabyAngel

    BabyAngel Newbie

    Joined:
    Nov 2, 2014
    Messages:
    19
    Likes Received:
    6
    only crawlers that obey robots.txt
     
  4. Holla1

    Holla1 Regular Member

    Joined:
    Aug 26, 2014
    Messages:
    230
    Likes Received:
    4
    a***ole

    _________________________________________________________________________________________
     
  5. Tobbe co

    Tobbe co Junior Member

    Joined:
    Sep 29, 2014
    Messages:
    171
    Likes Received:
    139
    Ehm, okey.. Sorry for answering your question ("Will this robots file block all search engines?")..

    You always this nice?

    The "User-agent: *" means this section applies to all robots (wildcard). The "Disallow: /" tells the robot that it should not visit any pages on the site.
    If you just want to block certain folder or categories you could use "Disallow: /category/", that will tell the spiders to not visit your urls looking like domain.com/category/xyz. While a single slash refers to the root of your domain.
     
    Last edited: Dec 18, 2014
  6. rob1977

    rob1977 Power Member

    Joined:
    Mar 25, 2013
    Messages:
    775
    Likes Received:
    676
    That escalated quick lol

    +1 for what tobbe says
     
  7. snarky

    snarky Junior Member

    Joined:
    Nov 21, 2009
    Messages:
    104
    Likes Received:
    58
    Since following robots.txt directives is voluntary and not all robots will obey, use an .htaccess file as well;

    Code:
    RewriteEngine On
    
    RewriteCond %{HTTP_USER_AGENT} (googlebot|bingbot) [NC]
    RewriteRule .* - [R=403,L]
    
    insert the robots you wish to bock.

    Now... can I be an a-hole as well?
     
  8. navinb

    navinb Junior Member

    Joined:
    Jun 25, 2009
    Messages:
    151
    Likes Received:
    85
    use both robot.txt and .htaccess. should block 99%
     
  9. tony_d

    tony_d Elite Member

    Joined:
    Jun 22, 2013
    Messages:
    2,583
    Likes Received:
    3,179
    Location:
    1600 Amphitheatre Parkway, Mountain View CA
    You certainly have a unique style...

    When someone answers my question, I typically thank them.
     
    • Thanks Thanks x 1