Why is Googlebot completely ignoring robots.txt

[mikie46]

So i added a directory to my robots.txt which basically says all files in my /support/ directory are off limits.

Today i notice Googlebot ignoring this request. First it reads my robots.txt

66.249.71.175 - - [01/Feb/2010:19:14:54 -0800] "GET /robots.txt HTTP/1.1" 200 5539 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Then is visits the knowledgebase articles in the directory i told it not to visit!!

66.249.71.175 - - [01/Feb/2010:19:14:55 -0800] "GET /support/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=160&nav=0%2C4%2C8 HTTP/1.1" 200 44787 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

WTF is up with that and since when does Google not understand;

Disallow: /support/

 

[mtravel13]

that`s strange !
is it just visiting the pages or caching it too ?
it might be that it takes a while to index the robots.txt behavior within G`s data centers and right now all this bot is doing is carrying that information over to google`s servers ..
one more interesting thing i found out was that
"Verifying Googlebot

You can verify that a bot accessing your server really is Googlebot by using a reverse DNS lookup, verifying that the name is in the googlebot.com domain, and then doing a forward DNS lookup using that googlebot name. This is useful if you're concerned that spammers or other troublemakers are accessing your site while claiming to be Googlebot.

"

are you sure it was googlebot and not some disguised bot trying to crawl your pages ?

 

[mikie46]

The ip is definitely Google. Iv see the same ip scattered throughout my server logs indexing other files and directories. Not sure why its not abiding by the rules.

Also, bots that ignore robots.txt usually dont read it. They just go ahead and index every directory. If its not Googlebot it wont read robots.txt usually.

 

[Dangazzm]

The ip is definitely Google. Iv see the same ip scattered throughout my server logs indexing other files and directories. Not sure why its not abiding by the rules.

Also, bots that ignore robots.txt usually dont read it. They just go ahead and index every directory. If its not Googlebot it wont read robots.txt usually.

Never looked this closely to how the Robot.txt and g00gle bot work but could it be possible what the poster above said?

That it will take the information back to be indexed and it will know in the future to NOT allow your directory? I would wait until the next time it comes back and see if THIS is how it works.