Importance of Robots.Txt?

achristopher

Registered Member
Joined
Mar 11, 2011
Messages
98
Reaction score
5
I get the basic idea of Robots.Txt, but I don't know exactly what to block and what to allow. I run a forum and I use a generic Robots.Txt file. However, when I check my Google webmasters account it appears that its blocking all sorts of pages, some of which I don't think should be blocked. But, at the same time my pages seem to be showing up in the SERPS with the correct Meta descriptions. Is it okay to just not have a Robots.Txt? All, it seems to do is cause problems.
 
A typical robots.txt file will have this
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/

Always allow your content,categories and menus. Another good point is to allow your images as Google will index them and if you've used alt tags you should get some extra traffic from the images alone.
 
I usually do this for wordpress :

User-agent: *
# Sensitive Folder
Disallow: /cgi-bin
Disallow: /wp-login.php
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /category/*/*
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: /*?
# sensitive files
User-agent: Googlebot
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.gz$
Disallow: /*.swf$
Disallow: /*.wmv$
Disallow: /*.cgi$
Disallow: /*.xhtml$
# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*
# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*
# sitemap
sitemap: "adress of the sitemap"
 
Back
Top