Noob question regarding robots.txt

Derviie

Regular Member
Joined
Oct 14, 2010
Messages
303
Reaction score
33
For my robots.txt, if I want it to just not index a certain directory, I just place it like so, right?:

User-agent: *
Disallow: /directory/

Well if that's the case, I've done it, and when I check my indexed pages it has index all those pages in that directory. Am I doing it wrong, or anything?
 
That is correct
Did you put that in right away? What I mean is - is there a chance robots came along before you put that in your robots file.

Also - I am assuming your robots.txt is in the root for your domain.

Strange if all the above is right and youre indexed.
 
Back
Top