Is my robots.txt file ok?

Discussion in 'Black Hat SEO' started by natewex, Aug 25, 2012.

  1. natewex

    natewex Power Member

    Joined:
    Sep 22, 2009
    Messages:
    607
    Likes Received:
    130
    Location:
    Ireland
    I am going to set up my products in a page called CATALOGUE.

    I really dont want google crawling this (as I am using wp robot to grab products fro amazon and as we all know thats considered duplicate content!) so have set up a robots txt file which looks like this:

    User-Agent: *
    Disallow:
    Disallow: /catalogue/


    Is this ok to keep google out of this directory and every page that may go into it? e.g. catalogue/red-widgets
     
  2. utuxia

    utuxia BANNED BANNED

    Joined:
    Feb 14, 2011
    Messages:
    673
    Likes Received:
    111
    yes. it is ok.
     
    • Thanks Thanks x 1
  3. natewex

    natewex Power Member

    Joined:
    Sep 22, 2009
    Messages:
    607
    Likes Received:
    130
    Location:
    Ireland
    Awesome, thanks very much :)