Googlebot visits main page and robots.txt hundreds times per day.
And it doesn't go anywhere else. (it goes, perhaps, but really very rarely so I've not seen that for months in my logs/stats).
I deleted my robots.txt at all and I don't see any improvements.
I have more than one million of pages on my site and more than 20k indexed in google. My website is about 3 years old and all the time indexed in google and giving traffic.
It has links to sitemap.html and sitemap.xml (more than 200 .xml) on main page. High DA/PA sites (more than 20, I guess) have links to my site.
And it doesn't go anywhere else. (it goes, perhaps, but really very rarely so I've not seen that for months in my logs/stats).
I deleted my robots.txt at all and I don't see any improvements.
I have more than one million of pages on my site and more than 20k indexed in google. My website is about 3 years old and all the time indexed in google and giving traffic.
It has links to sitemap.html and sitemap.xml (more than 200 .xml) on main page. High DA/PA sites (more than 20, I guess) have links to my site.