GSC Error (Failed.Robots.txt)

Lifeh18 said:

Hi guys, i just tried to manually index a page but im getting the error ''Failed: Robots.txt unreachable''. I havent done anything to cause this and it just started on is own. Anyone experienced this or have any tips on what to do.

Anyone? This error is driving me crazy because I can't understand how it just happened

Click to expand...

PHPInjected said:

Who is your DNS setup through? I'd check there.

Click to expand...

Lifeh18 said:

I contacted my hosting and they said there are no changes. This issue started all of a sudden

Click to expand...

PHPInjected said:

Who is your provider?

Click to expand...

Lifeh18 said:

What could I possibly ask them? I asked them if any ips are blocked and they said no. Everything is fine. Now I'm not sure what else could be the problem.

Click to expand...

• Use curl (or similar program) to fetch the robots.txt file with a user-agent of Googlebot to see if the site might have some firewall rules on that file that are blocking Google.
• Grep the logs to see if Googlebot has fetched the robots.txt file. That would tell you if they are getting a 200 response or something like a 4xx (client error) or 5xx (server error). If you don't see any hits from Google on robots.txt in the logs, then they might be getting blocked at another layer, like Cloudflare.

PHPInjected said:

Do you have any caching plugins enabled? Are you using Litespeed?

I'd ask them to double check the blocked IPs again.

Also, what does your robot.txt file look like? Scroll through it and see if it's disallowing Googlebot to crawl.

I did some extra digging around for you and found this:

A couple of places to search for clues:

• Use curl (or similar program) to fetch the robots.txt file with a user-agent of Googlebot to see if the site might have some firewall rules on that file that are blocking Google.
• Grep the logs to see if Googlebot has fetched the robots.txt file. That would tell you if they are getting a 200 response or something like a 4xx (client error) or 5xx (server error). If you don't see any hits from Google on robots.txt in the logs, then they might be getting blocked at another layer, like Cloudflare.

Outside of that, it seems to me that it's a server/DNS issue with your hosting company and will most likely work itself out (I know how that sounds, but it's true).

Click to expand...

Lifeh18 said:

My hosting says everything is fine on their side, google ips are already whitelisted. So it can't be my hosting issue

Click to expand...

• Check Robots.txt File:
  • Ensure that the robots.txt file exists on your server and is accessible at http://yourdomain.com/robots.txt.
  • Verify the file's permissions to ensure it is readable by everyone (usually permissions set to 644).
• Server Configuration:
  • Check your server settings to ensure there are no restrictions or errors that might prevent the robots.txt file from being served.
  • Look into server logs to identify any potential issues or errors related to the robots.txt file.
• DNS Issues:
  • Ensure there are no DNS issues that might be preventing the tool from resolving your domain correctly.
• Firewall/Hosting Provider:
  • Sometimes firewalls or security settings on your hosting provider’s side might block access to certain files. Ensure your hosting provider is not blocking access to the robots.txt file.
• Temporary Glitches:
  • Occasionally, this issue might be due to a temporary glitch with your hosting provider or network. Try accessing the file after some time.
• Crawler/Indexing Tool:
  • Verify that the problem is not specific to the tool you're using. Try accessing the robots.txt file using a different tool or manually through a browser.

hideath said:

are you using CF?
did you double check the htaccess file?
are you sure you're using the right web url base in the GSC (www, https, etc...)?

pm me the website url, i'll make a quick test

Click to expand...