Blocking Google with robots.txt?

Discussion in 'Black Hat SEO' started by samiejg, Dec 29, 2013.

  1. samiejg

    samiejg Senior Member

    Joined:
    Dec 14, 2013
    Messages:
    1,151
    Likes Received:
    71
    Say if I have a huge website with many articles being duplicate content.. if I use a robots.txt file to disallow Google from those particular articles, will they find out or penalize me in any way? Or even if I was doing even something worse?
     
  2. peetrike

    peetrike Power Member

    Joined:
    Aug 19, 2012
    Messages:
    585
    Likes Received:
    219
    Location:
    Estonia
    You can easily set noindex to these posts. They may only do this with a manual review.

    Cheers
     
    • Thanks Thanks x 1
  3. tahworld

    tahworld Regular Member

    Joined:
    Aug 16, 2013
    Messages:
    457
    Likes Received:
    393
    Location:
    ✔✔✔✔✔✔✔
    Use htaccess

    RewriteEngine On
    RewriteCond %{HTTP_USER_AGENT} Googlebot
    RewriteRule . - [F,L]

    place in the folder you want to block

    a safer way is by IP, but you would have to find google's IP's first.
     
  4. stugz

    stugz Junior Member

    Joined:
    Apr 14, 2013
    Messages:
    154
    Likes Received:
    34
    Nslookup followed by reverse nslookup will tell you for sure an IP address is from Google.
     
    • Thanks Thanks x 1