Robots.txt

S

simmonsmike7

BANNED
Joined
Jun 29, 2008
Messages
122
Reaction score
21
I want to make this entire directory on my site not showing to any search engine, and I think you do this by adding a robots.txt file.

1. Am I right?

2. If so, where do I get this file?

3. And finally, where do I place it.

I have the directory installed on my website such as: http://www.mywebsite.com/directory

Thank You.
 
Advertise on BHW
You can easily create a robots.txt using an online generator. Try the following listed here:
Code: 
http://www.mcanerin.com/en/search-engine/robots-txt.asp
 
Don't rely on just a Robots.txt 4 this as it will fail
 
If you don't want the spiders to find something then do not
have any internal or external links to it. All spiders do is
follow links.
 
Just blacklist all spiderIP's, their are websites who got lists of over 10.000 IP's from all sort of spiders. You could also just make everything in javascript (text output) and with some encryption. Spiders don't execute javascript so that means they won't see the text.
 
Also be aware that people can read the robots.txt file. It's a common place to look if people are trying to find hidden pages on your site.
 
and another thing to remember - robots.txt files only stops robots that obey the rules.
Any scripted bot that doesn't obey the txt file will still stamp all over your site.
 
• use robots.txt - disallow all known robots heck even block voltes V, megatron and Voltron
• php header redirect - when the script finds the referral/ IP is from google/ yahoo etc
redirect them
• put this between <head></head>
<meta name="robots" content="noindex, nofollow" />
• put it in frame
• put a username and password (create an image for the visitors on how and what the username and passwords are)

whatelse?
 
first make a robots.txt search google robots.txt generator
second make it password protected with .httacsess and you will be fine (=
 
jb01 said:
If you don't want the spiders to find something then do not
have any internal or external links to it. All spiders do is
follow links.
Click to expand...

jb01 is correct. This is all I do (or should I say, don't do) to protect my pages.
No links means the spiders dont know about it.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Networking Downloads

Marketplace

Content / Copywriting Hosting Images, Logos & Videos Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design Misc

Making Money

Affiliate Programs Hire a Freelancer Making Money Pay Per Click Site Flipping

BlackHatWorld

BHW Merch Forum Suggestions & Feedback Introductions Lounge Dispute Resolution

Other

Domain Name Forum IM Journeys Web Hosting Newsletter
Back
Top