1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Blocking Google from crawling certain folder.

Discussion in 'Cloaking and Content Generators' started by Markthedude, Sep 5, 2011.

  1. Markthedude

    Markthedude Power Member

    Joined:
    Feb 26, 2010
    Messages:
    572
    Likes Received:
    266
    Occupation:
    Entrepreneur
    Location:
    United States
    Ok, so I'm using a simple .php code to cloak my affiliate links.

    I have a subfolder setup on my server called "go" for example that has all the php files in it and nothing else.

    I was reading some posts on here and didn't see a clear answer. Should I make a simple robot.txt to disallow search engines from crawling it?

    Does it hurt my SERPS letting google crawl it and having them find out that I'm an affiliate? That's my main question and only reason I would block G. from crawling it.

    Thanks for any help!
     
  2. shadowrizer

    shadowrizer Newbie

    Joined:
    Feb 8, 2011
    Messages:
    10
    Likes Received:
    0
    What I would do is use htaccess to either block Google or have Google see something different by rewriting the URL.
     
  3. machines-in-motion

    machines-in-motion Junior Member

    Joined:
    Aug 11, 2010
    Messages:
    129
    Likes Received:
    46
    Occupation:
    Engineer
    Location:
    Here and there
    Google respects robots.txt. Just write in that bots aren't allowed to access those pages.
     
  4. losille

    losille Junior Member

    Joined:
    Feb 22, 2011
    Messages:
    109
    Likes Received:
    95
    Look over this page there a couple of ways. I use meta tag.
    Code:
    http://antezeta.com/news/avoid-search-engine-indexing
     
  5. typeslowly

    typeslowly Registered Member

    Joined:
    Nov 30, 2008
    Messages:
    61
    Likes Received:
    9
    Location:
    United States
    This. But be aware some users will look at your robots.txt to try to find hidden shit.