Basic SEO practices for newbies:

thinksmarty said:

I am still kind of confused as to what the file actually does.

Click to expand...

Here are a couple of resource links to help you out and better explain what robot.txt. files are and do:

http://www.robotstxt.org/robotstxt.html

http://www.webconfs.com/what-is-robots-txt-article-12.php

http://www.outfront.net/tutorials_02/adv_tech/robots.htm

Warning Tip: Do NOT play with these files until you know what you are doing with them or you will end up doing more damage than good with them.

Just came across this excellent thread and wish I had found it this time last year. This site never ceases to amaze me. Thanks to all contributors. There are some great tips in here!

aftershock2020 said:

thinksmarty said:

I am still kind of confused as to what the file actually does.

Click to expand...

Here are a couple of resource links to help you out and better explain what robot.txt. files are and do:

http://www.robotstxt.org/robotstxt.html

http://www.webconfs.com/what-is-robots-txt-article-12.php

http://www.outfront.net/tutorials_02/adv_tech/robots.htm

Warning Tip: Do NOT play with these files until you know what you are doing with them or you will end up doing more damage than good with them.

Click to expand...

@thinksmarty -
Robots.txt is just a file that tells crawlers, spiders, and bots what sections of your website you want them to check out, (or more specifically which ones you don't). As far as SEO is concerned the value is in directing the crawlers to the portions of your site that you're focused on having indexed, (by restricting them from the pages you really don't want to have in the index).

For the average websites that most people have it isn't really necessary to use one, but if it doesn't exist then you'll get a 404 error in your server logs for everytime a crawler visits your website. For that reason I suggested to at least have a blank robots.txt file. All the crawler is really looking for in the robots.txt is for any thing you want to restrict them from. If it's an empty file then there are no restrictions and the crawler will look at whatever it wants on your site, but having it will prevent the 404 errors for it. If you do have pages or folders that you don't want indexed then placing some restrictions in the robots.txt is one way to deal with that.

The resources that aftershock linked to in his reply can help you understand how to set up those restrictions and when you might want to. Take his warning seriously though, if you set the restrictions up incorrectly then you could have very negative effects on the indexing of your site. Set it up wrong and your website might not get indexed at all.

If you don't have a really good reason for using it, then you're probably better off just leaving it as an empty file. I've only had a couple websites which had a need to use one, the bulk of websites I've had over the years didn't need to use one. In those cases I simply include a blank robots.txt just to prevent 404 errors for it.

---
Same thing with a favicon. It isn't really needed for most sites, but if you don't have one then you'll get a 404 error in your log for every visitor to your site. Simply including a favicon in your puplic folder will prevent all those error codes. A favicon will have ZERO effect for SEO, but it's so simple to add one there isn't really any reason not to.

Perfect post for new people. Everybody new seems to get so caught up in linkbuilding etc, they forget about decent on page SEO

i'm new here, thank you for that post. It helped me alot

Newbie here, thank you for the helpful info!

good share,thanks

Thanks it looks like great advce

Got2Know said:

Just came across this excellent thread and wish I had found it this time last year. This site never ceases to amaze me. Thanks to all contributors. There are some great tips in here!

Click to expand...

Indeed its like a whole new world

Got2Know said:

Just came across this excellent thread and wish I had found it this time last year. This site never ceases to amaze me. Thanks to all contributors. There are some great tips in here!

Click to expand...

Second that!

Thanks again for all the tips!

Thank you !!

Mama mia,...wonderful sources...

Yes, nice for newbies in seo. It would be perfect start to drive seo for a website.

Hey guys, just got through helping out another post in the forum with a dating site issue. I don't normally give out this type of info but it suits this area as much as it did the actual original poster's thread.

Take advantage and put this knowledge to use, as I've stated, I don't reveal this type of detail often.

http://www.blackhatworld.com/blackhat-seo/black-hat-seo/345683-there-light-end-tunnel.html

This is a really helful post, no one birn with this kind of knoledge and is nice to find it in one place.

the basics are the foundation of knowledge acquired, greywolf your teh man

First, thanks Greywolf for wonderfull post. It's clear me a lot of questions.
Second, please, if somebody explain me what kind of correlation have robots.txt and a favicon.ico?

dan777 said:

First, thanks Greywolf for wonderfull post. It's clear me a lot of questions.
Second, please, if somebody explain me what kind of correlation have robots.txt and a favicon.ico?

Click to expand...

No correlation at all. lol

The only reason I mentioned those is that if you don't have them then your server logs will be filled with 404 errors because every time a visitor comes to your site the browser will request the favicon.ico and every time a crawler comes it will request the robots.txt.

As far as SEO is concerned neither are necessary. (Well the robots.txt can be helpful in some cases, but for most people it isn't necessary). The only reason I even mentioned them at all in the original post is because having them helps keep your server logs cleaner.

If it confuses you at all then just ignore any mention of robots.txt or favicon.ico in this thread. Like I said, they have nothing to do with the SEO of your site at all.

Hope that clears it up. :cool2:

Thanks, good info, great post...to favorites.