1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Is my robots.txt file ok?

Discussion in 'Black Hat SEO' started by natewex, Aug 25, 2012.

  1. natewex

    natewex Power Member

    Joined:
    Sep 22, 2009
    Messages:
    602
    Likes Received:
    127
    Location:
    Ireland
    I am going to set up my products in a page called CATALOGUE.

    I really dont want google crawling this (as I am using wp robot to grab products fro amazon and as we all know thats considered duplicate content!) so have set up a robots txt file which looks like this:

    User-Agent: *
    Disallow:
    Disallow: /catalogue/


    Is this ok to keep google out of this directory and every page that may go into it? e.g. catalogue/red-widgets
     
  2. utuxia

    utuxia BANNED BANNED

    Joined:
    Feb 14, 2011
    Messages:
    673
    Likes Received:
    111
    yes. it is ok.
     
    • Thanks Thanks x 1
  3. natewex

    natewex Power Member

    Joined:
    Sep 22, 2009
    Messages:
    602
    Likes Received:
    127
    Location:
    Ireland
    Awesome, thanks very much :)