XML Sitemap for 60K+ pages

sirron

Newbie
Joined
Apr 5, 2010
Messages
2
Reaction score
0
I have a site with 60K+ pages and am looking to EXclude URLs from my sitemap, or prevent Google indexing, if the URLs contain certain keywords. Using disallow on robots.txt doesn't work because the URLs I'm looking to exclude are not in a particular directory. Any thoughts much appreciated.
 
your robots.txt file should be enough to limit any URL within your domain.

you dont have to just use disallow either. Have a look in some code forums for help on it and advise
 
use an .htaccess file and add basic authentication to sites you want to protect. thats simple and works for every visitor (bots and normal users). you can use mod_rewrite rules for that. setup is simple and straight forward.
 
Last edited:
thanks for the help here. I'm likely going to take another route which is to dynamically insert the Noindex tags. Going to use something like this to add noindex tags on pages that matched the criteria for being excluded (in PHP)

if (preg_match($regex, $_SERVER['REQUEST_URI']))
{
header('X-Robots-Tag: noindex, nofollow, noarchive');
}
 
Back
Top