Robots.txt help

Hi

I need help with this, currently my Robots.txt file is setup in this order as i dont want my website to be crawled in other directories other than the important ones?

User-agent: *
Disallow:
Disallow: /cgi-bin/
Disallow: /Admin/
Disallow: /images/
Disallow: /Account/
Disallow: /bin/
Disallow: /ckeditor/
Disallow: /Controls/
Disallow: /Css/
Disallow: /Emailing/
Disallow: /feeds/
Disallow: /js/
Disallow: /obj/
Disallow: /Parent/
Disallow: /slider/
Disallow: /*.PDF$
Disallow: /*.jpeg$
Disallow: /*.exe$
Disallow: /asp/

My website tree structure looks like this:


Can you help?

I don't understand the question. That's the way a robots file works. Do you want to know if you've missed anything?

You should remove the first "Disallow:" line as it allows the crawler access to all directories and files.

I always make my images "Allowed" because the search engines will index the images and if you've used your image alt tags correctly by using your target KW, you can gain quite a bit of extra traffic from them.

I dont want the bots to crawl all my content which is unnecessary or useless info! As you can see from the image i uploaded a screenshot of my website's FTP. Obviously there is a default.asp ......! I want to know in the image of my FTP which ones should i block from Google, Yahoo and Bing Crawlers for my website still to be indexed correctly?

Thx for the current comments!!!

Why do you want to block those pages from the crawlers? It seems more likely you harm yourself here.

pctank said:

I want to know in the image of my FTP which ones should i block from Google, Yahoo and Bing Crawlers for my website still to be indexed correctly?

Click to expand...

From this image it's impossible to say which areas of your site should be restricted from indexing by search engines.

The point is that, there must be some reasons why you need to restrict certain pages of your site from search engine bots. These reasons can be:

- privacy: you may want to hide pages that have your clients' info, info about your company alpha products, etc;
- duplicate content: pages that have content that can be found somewhere else, should be hidden from indexation;
- little of no value to site visitors: pages that are not useful to your site visitors ('thank you' pages, pages under development, etc.) should not be accessible to search engine crawlers.

So, if you have the reasons listed above, you need to hide the corresponding pages of your site from search engine bots.

If you don't have any reason to hide your site pages, let search engines index them. To improve the process of indexation:

- generate an .xml sitemap: it will help search engines better crawl your site and its pages;
- check if your site pages are properly interlinked.

Hope, I have answered your question.

OK Thanks for that guys!!
This is my Google issue as this has been generated by Google ERROR!

Bing:


Very frustrating!!!

Those are 302 errors in the images which can't be fixed by your robots.txt. You're going to want the .htaccess file for those and change them to 301 redirects (don't just change the 2 to a 1, look up how to write them, it's easy.)

Are those errors the original problem of this thread? I'm not quite sure what problem you're trying to fix here. the GWMT errors or your robots.txt?

"Are those errors the original problem of this thread? I'm not quite sure what problem you're trying to fix here. the GWMT errors or your robots.txt?"

Hi, thank you!!! Well both was my issue but the 302 error is more important! I will look into this as this is more of an issue!

You better use .haccess for permissions/restrictions. It has way more power then robots.txt

Thank you ContentLockPro! The advise highly appreciated!!!