what about this robots.txt for wordpress?

iamok22

Regular Member
Dec 24, 2009
296
4
is it good in your opinion?

User-agent: *
# disallow all files in these directories
Disallow: /cgi-bin/
Disallow: /z/j/
Disallow: /z/c/
Disallow: /stats/
Disallow: /dh_
Disallow: /about/
Disallow: /contact/
Disallow: /tag/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /contact
Disallow: /manual
Disallow: /manual/*
Disallow: /phpmanual/
Disallow: /category/
User-agent: Googlebot
# disallow all files ending with these extensions
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.gz$
Disallow: /*.wmv$
Disallow: /*.cgi$
Disallow: /*.xhtml$
# disallow all files with ? in url
Disallow: /*?*
# disable duggmirror
User-agent: duggmirror
Disallow: /
# allow google image bot to search all images
User-agent: Googlebot-Image
Disallow:
Allow: /*
# allow adsense bot on entire site
User-agent: Mediapartners-Google*
Disallow:
Allow: /*

Sitemap: http://www.**********.com/sitemap.xml
 
I dont think dissalowing tags, category, about and contact are good for WP. Tags and category are good for relevancy, about and contact are good for credibility.

Plugins is the one directory you dont want bots to snoop around and even that only if you have some shady plugins
 
I dont think dissalowing tags, category, about and contact are good for WP. Tags and category are good for relevancy, about and contact are good for credibility.

Plugins is the one directory you dont want bots to snoop around and even that only if you have some shady plugins

about tags: it is made in that way to avoid duplicate content...anyway i am not sure about this.some pople say that i should use tags and some not.

about Disallow: /category/ it is made to avoid duplicate content and since i give a specific name to each category i think it won't be a problem.
this will only prevent the bot to access the "/category/" not gategories with a specific name assigned.

let me know if i am wrong...
 
you are right about "category" and "tag" permalinks creating duplicate content; but, I think you best use some sort of plugin to manage that problem,and not robots.txt.
sure,best use robots.txt to "hide" some of the directories but think; if for any reason you change the permalink structure you'll have to change robots.txt, etc, meanwhile the plugin takes care by itself about those problems.
 
Back
Top
AdBlock Detected

We get it, advertisements are annoying!

Sure, ad-blocking software does a great job at blocking ads, but it also blocks useful features and essential functions on BlackHatWorld and other forums. These functions are unrelated to ads, such as internal links and images. For the best site experience please disable your AdBlocker.

I've Disabled AdBlock