1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Noob question regarding robots.txt

Discussion in 'Black Hat SEO' started by Derviie, Oct 23, 2010.

  1. Derviie

    Derviie Regular Member

    Oct 14, 2010
    Likes Received:
    For my robots.txt, if I want it to just not index a certain directory, I just place it like so, right?:

    User-agent: *
    Disallow: /directory/

    Well if that's the case, I've done it, and when I check my indexed pages it has index all those pages in that directory. Am I doing it wrong, or anything?
  2. Spaceman

    Spaceman Regular Member

    Aug 8, 2009
    Likes Received:
    That is correct
    Did you put that in right away? What I mean is - is there a chance robots came along before you put that in your robots file.

    Also - I am assuming your robots.txt is in the root for your domain.

    Strange if all the above is right and youre indexed.