Noob question regarding robots.txt

Discussion in 'Black Hat SEO' started by Derviie, Oct 23, 2010.

  1. Derviie

    Derviie Regular Member

    Joined:
    Oct 14, 2010
    Messages:
    284
    Likes Received:
    33
    For my robots.txt, if I want it to just not index a certain directory, I just place it like so, right?:

    User-agent: *
    Disallow: /directory/

    Well if that's the case, I've done it, and when I check my indexed pages it has index all those pages in that directory. Am I doing it wrong, or anything?
     
  2. Spaceman

    Spaceman Regular Member

    Joined:
    Aug 8, 2009
    Messages:
    435
    Likes Received:
    53
    That is correct
    Did you put that in right away? What I mean is - is there a chance robots came along before you put that in your robots file.

    Also - I am assuming your robots.txt is in the root for your domain.

    Strange if all the above is right and youre indexed.