I am starting to build some autoblogs and I am a bit concerned about how to protect myself from big G. crawls and robots.. I know this crawls and robots check everything in your page, including your plugins subdirectory. Whats the best way to protect myself from this check? Because once the spider get to my plugins they will know through the names that it is an autoblog... Does robots.txt works? I mean, will big G. respect it? or I should change each plugins subdirectories name and respectives php files? I dont know if I am being paranoid just would like some suggestions pls.
I think you're just being a little paranoid, personally I have lots of autoblogs and never cared about it...but if you really insist, create in Notepad a file named ".htaccess" and upload it to main folder of your wordpress installation: Code: < Directory /wp-content/plugins/> Order deny,allow Deny from all </Directory> You can also (if you already haven't) upload a robots.txt file with the following: Code: Sitemap: /sitemap.xml User-agent: * Disallow: /wp-content/cache/ Disallow: /wp-content/themes/ Disallow: /wp-content/plugins/ Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-login.php
I created the .htaccess file like you said and uploaded it and it gave me a 500 Internal Server Error, everything came back to normal after I deleted the .htaccess. Did I do anything wrong? My hosting is Dreamhost Linux Based so I think it is not the problem.
You can also put .htaccess with only this: Code: Order allow,deny Deny from all In the directory "/wp-content/plugins/" - where you keep the plugins. For some reason the first version doesn't work always on some hosts...
If you use htaccess and robots.txt you'll prevent good bots like Google's from crawling the directories you list. So just do what was mentioned above and you should be set.
