Expert Advice Needed on Robots.txt

Hello BHW,

I have portal website that is hosted in amazon with apache.
I want to block some urls from Google so I have included those into robots.txt

User-agent: *
Disallow: /load_cal_ajax/
Disallow: http://prod.example.com
Disallow: http://example.com/search/get_cat_by_city/
Disallow: http://example.com/search/get_cat_by_city/Goa
Disallow: http://example.com/search/load_venue_by_cat/Bangalore
Disallow: http://example.com/search/load_venue_by_cat/Hyderabad
Disallow: http://example.com/search/load_venue_by_cat/Goa
Disallow: http://example.com/search/load_venue_by_cat/Mumbai
Disallow: http://example.com/welcome/load_cities
Disallow: http://example.com/welcome/load_cities/Bangalore
Disallow: http://example.com/welcome/load_cities/Goa
Disallow: http://example.com/welcome/load_cities/Pune
Disallow: http://example.com/pagenotfound
Disallow: http://example.com/m/

Is it right way to block urls to be index. I uploaded this file a month ago but these urls keep indexing in google.

You'd be better off placing the <NOINDEX> tag in the header of the pages you don't want indexed.

That way they have to read the tag instructing them not to index the page because it's on the page itself. So you're not relying on them to read and comply with your robots.txt file.

That being said, in theory at least, Gbot should normally always look for and abide by the instructions in your robots file, so I'm surprised that these URLs are being indexed

You don't need to include the full url in robots.txt, so your first directive is actually correct. For the others simply remove http://example.com and start with a leading /.

Additionally, if you're using webmaster tools, you can remove the urls from there.

the interesting part is there is option call crawl error >> blocked urls.
It shows 0 blocked urls?

Full URL is not required for the robots.txt file you can copy-paste below coding in your robots.txt file or you can manually use Nofollow, NoInddex meta tags on the particular pages.


User-agent: *
Disallow: /load_cal_ajax/
Disallow: /search/get_cat_by_city/
Disallow: /search/get_cat_by_city/Goa
Disallow: /search/load_venue_by_cat/Bangalore
Disallow: /search/load_venue_by_cat/Hyderabad
Disallow: /search/load_venue_by_cat/Goa
Disallow: /search/load_venue_by_cat/Mumbai
Disallow: /welcome/load_cities
Disallow: /welcome/load_cities/Bangalore
Disallow: /welcome/load_cities/Goa
Disallow: /welcome/load_cities/Pune
Disallow: /pagenotfound
Disallow: /m/

Thanks to all of you but I want to block it from Robots.txt
And is it strange issue, Have anyone had this kind of issue ever with 0 blocked urls in Google webmaster Tool.

`ere u go mate

http://www.mcanerin.com/EN/search-engine/robots-txt.asp

lmxftw said:

`ere u go mate

http://www.mcanerin.com/EN/search-engine/robots-txt.asp

Click to expand...

This is quite useful to make robot.txt. I have already created and uploaded robots.txt file but how is it still indexing.

Use this on the necessary pages:

Treeofl1 said:

Use this on the necessary pages:

Code:

<meta name="robots" content="noindex,nofollow">

Click to expand...

What about directories. I want to block directories.