the1aquemini
Junior Member
- Jul 7, 2020
- 101
- 48
If I just have this in my robots.txt file I will be allowing google to still crawl my site right? Or do I need to have it like the following examples:
Is this robots.txt file blocking google?
- Thread starter the1aquemini
- Start date
If I just have this in my robots.txt file I will be allowing google to still crawl my site right? Or do I need to have it like the following examples:
Ashk881
Elite Member
- Aug 24, 2021
- 1,695
- 2,072
In the first one you don't state what routes are allowed at all. Googlebot might take that as the equivalent of not allowing anything and not crawl any page.
You will need to have the second example if you want Google to crawl all your pages.
the1aquemini
Junior Member
- Jul 7, 2020
- 101
- 48
I mean surely it wouldn't though? Does anybody actually know for sure?
https://technicalseo.com/tools/robots-txt/
I've tested it here and it reports fine, and my site is still indexed and ranking normally and I have had that file for a bit less than a week.
- Apr 4, 2014
- 6,454
- 4,861
Have you ever thought of using google????
User-agent: * - means all agents
Allow: / - means allow the entire site
the1aquemini
Junior Member
- Jul 7, 2020
- 101
- 48
Yes I googled the question originally, I am aware I can change it and add Allow: /
My question is, is it actually ever necessary and does not having it ever prohibit Google from crawling anything? As when there is no direction, google will surely automatically crawl the site? It makes zero sense to me that it would take it as an instruction to not crawl...
- Apr 4, 2014
- 6,454
- 4,861
I see no point of
User-agent: *
Allow: /
Think of it as laws... do I lawmakers come out and say " you are allowed to do this and that"? NO
the1aquemini
Junior Member
- Jul 7, 2020
- 101
- 48
Yes that is my point... So unless me specifically specifying the law to not crawl X then it should automatically crawl it regardless. So the first listed example should work equally well.
Ashk881
Elite Member
- Aug 24, 2021
- 1,695
- 2,072
Well the validator works because it's not an invalid robots.txt file. It just doesn't explicitly tell googlebot that you are allowing it to crawl the entire site.
The robots.txt is only a suggestion for crawling rules. It's upto the crawler whether it wants to obey it or not. Googlebot does seem to follow it.
Now not having an explicit allow flag might imply googlebot can still crawl your site. But it will stop working once Devs of googlebot configure it so it only crawls if you explicitly allow it to crawl your site.
Why do you even care about this tho? You just want your robots.txt to be as explicit as possible. It isn't like adding an extra line to your robots.txt is hurting you or something.
Ashk881
Elite Member
- Aug 24, 2021
- 1,695
- 2,072
That's not how it works. A bot isn't a human.
You might have the intuition and common sense that if a lawmaker doesn't explicit tell you something is legal, you just assume it is. (which isn't true a lot of times too btw)
Googlebot doesn't have any of that common sense tho. It's software and it only understands configuration files.
If the config file tells it that "crawl pages that are not in an Allow: but are not in Disallow: either". The bot will follow that.
But if the config file tells "only crawl pages that are in an Allow:", googlebot will follow that and then none of your pages will get crawled in the future because you didn't explicitly tell it to.
Ashk881
Elite Member
- Aug 24, 2021
- 1,695
- 2,072
That's the thing. You don't specify the law to not crawl X. You only suggest Googlebot to not crawl X. It still can do it if it wants to. And you can't control that.
You don't control googlebot. It's config files set by it developers do.
Now, i don't know about you but when I am making a suggestion, I usually think it's a good idea to be as explicit about my suggestion as I can be. Especially when it's a piece of software.
Googlebot tries a lot to minimize it's crawl budget anyways. It dynamically decides what to crawl and what not to. Who knows, if it's algorithms decide to not crawl any of your pages if it doesn't see a single
Allow: flag in your robots.txt 
the1aquemini
Junior Member
- Jul 7, 2020
- 101
- 48
I am asking for the sake of clarity, I appreciate the answers from all but it seems to only be speculation at best from anyone...
Ashk881
Elite Member
- Aug 24, 2021
- 1,695
- 2,072
I still don't get why are you thinking so much about a single line in a file.
It's regarded an industry wide good practice to be as explicit with the robots file as you can for the sake of clarity for the bots.
Point about being explicit in the robots might as well be speculation, but it's harmless. There's no harm in adding a couple extra lines to it.