1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Blocking Google with robots.txt?

Discussion in 'Black Hat SEO' started by samiejg, Dec 29, 2013.

  1. samiejg

    samiejg Senior Member

    Joined:
    Dec 14, 2013
    Messages:
    1,134
    Likes Received:
    69
    Say if I have a huge website with many articles being duplicate content.. if I use a robots.txt file to disallow Google from those particular articles, will they find out or penalize me in any way? Or even if I was doing even something worse?
     
  2. peetrike

    peetrike Power Member

    Joined:
    Aug 19, 2012
    Messages:
    585
    Likes Received:
    218
    Location:
    Estonia
    You can easily set noindex to these posts. They may only do this with a manual review.

    Cheers
     
    • Thanks Thanks x 1
  3. tahworld

    tahworld Regular Member

    Joined:
    Aug 16, 2013
    Messages:
    457
    Likes Received:
    393
    Location:
    ✔✔✔✔✔✔✔
    Use htaccess

    RewriteEngine On
    RewriteCond %{HTTP_USER_AGENT} Googlebot
    RewriteRule . - [F,L]

    place in the folder you want to block

    a safer way is by IP, but you would have to find google's IP's first.
     
  4. stugz

    stugz Junior Member

    Joined:
    Apr 14, 2013
    Messages:
    154
    Likes Received:
    34
    Nslookup followed by reverse nslookup will tell you for sure an IP address is from Google.
     
    • Thanks Thanks x 1