how to hide robots.txt

G

glasshalffull

Registered Member

May 30, 2008

77

50

• Aug 3, 2009

• #1

when i add robots.txt to the end of my url i pull up some pages that are disallowed

how do i stop that robots.txt from showing my hidden pages? I remember a thread aobut this in here but can't find it?

thanks

 

Kingfresh

Regular Member

Jul 8, 2009

369

297

• Aug 3, 2009

• #2

robots is just for spiders (google spider etc.)
http://en.wikipedia.org/wiki/Robots_exclusion_standard

to hide something from public - use .htaccess
http://en.wikipedia.org/wiki/.htaccess

 

G

glasshalffull

Registered Member

May 30, 2008

77

50

• Aug 3, 2009
• Thread Starter Thread Starter

• #3

thanks King

 

GridPanel

Power Member

Jr. VIP

Jul 25, 2009

669

233

• Aug 3, 2009

• #4

Remember that you can fake your way into your robots.txt (especially if you only use user-agent checking), so don't leave anything in there you never want people to see.

 

kouxx

Junior Member

Aug 2, 2008

160

39

• Aug 3, 2009

• #5

you can use htaccess as said above. thats when you want to use a password.

if you just dont want the spiders to crawl you pages you can just use noindex in your meta tags. use the searchengine of you choice to search for "noindex meta tag"

edit: when using meta tags you DO NOT need to edit yout robots.txt

 

Kingfresh

Regular Member

Jul 8, 2009

369

297

• Aug 3, 2009

• #6

glasshalffull said:

thanks King

Click to expand...

you are welcome

 

G

glasshalffull

Registered Member

May 30, 2008

77

50

• Aug 3, 2009
• Thread Starter Thread Starter

• #7

basically i just don't want people that know how to use the robots.txt to be able to access any pages I don't want seen, so will htaccess be the easiest?

thanks

kouxx said:

you can use htaccess as said above. thats when you want to use a password.

if you just dont want the spiders to crawl you pages you can just use noindex in your meta tags. use the searchengine of you choice to search for "noindex meta tag"

edit: when using meta tags you DO NOT need to edit yout robots.txt

Click to expand...

 

kouxx

Junior Member

Aug 2, 2008

160

39

• Aug 4, 2009

• #8

the easiest way would be not to upload a robots.txt
you do not need the file at all.
and for the SE's: as long as you to don't link to the sites, they will not crawl it. If you give a link to someone else and that person posts the link in a fourm or something you should delete the page anyway.