normally for any site, we do use robots.txt to block spiders. do i have to disallow my wp-content and wp-include folder? Disallow: /wp-content/ Disallow: /wp-include/ and can we do it like this? Disallow: /subdomain/wp-content/ or do we have to add a seperate robots.txt file for each subdomain?
If you are having latest wordpress 2.8 + then there is already a blank index.php, so if you type yourdomain.com/wp-content then it will show blank page. No need to use robots.txt I dont know about the old version. But still you can insert a blank index.php in to the folder if its is already not there. Hope it helped
I've included a blank html page was just wondering if do i need to add disallow in the robots im running 2.7.1
