GSC Error (Failed.Robots.txt)

[Lifeh18]

Hi guys, i just tried to manually index a page but im getting the error ''Failed: Robots.txt unreachable''. I havent done anything to cause this and it just started on is own. Anyone experienced this or have any tips on what to do.

Anyone? This error is driving me crazy because I can't understand how it just happened

 

[Deep9889]

First of check firewall if its blocking the robot.txt url also check url accessibility make sure its not giving 404.

 

[Lifeh18]

It's all working there.

It's only when I try to index then also says "can't fetch robots.txt"

 

[PHPInjected]

Hi guys, i just tried to manually index a page but im getting the error ''Failed: Robots.txt unreachable''. I havent done anything to cause this and it just started on is own. Anyone experienced this or have any tips on what to do.

Anyone? This error is driving me crazy because I can't understand how it just happened

Who is your DNS setup through? I'd check there.

 

[Lifeh18]

Who is your DNS setup through? I'd check there.

I contacted my hosting and they said there are no changes. This issue started all of a sudden

 

[PHPInjected]

I contacted my hosting and they said there are no changes. This issue started all of a sudden

Who is your provider?

 

[Lifeh18]

Who is your provider?

What could I possibly ask them? I asked them if any ips are blocked and they said no. Everything is fine. Now I'm not sure what else could be the problem.

 

[PHPInjected]

What could I possibly ask them? I asked them if any ips are blocked and they said no. Everything is fine. Now I'm not sure what else could be the problem.

Do you have any caching plugins enabled? Are you using Litespeed?

I'd ask them to double check the blocked IPs again.

Also, what does your robot.txt file look like? Scroll through it and see if it's disallowing Googlebot to crawl.

I did some extra digging around for you and found this:

A couple of places to search for clues:

• Use curl (or similar program) to fetch the robots.txt file with a user-agent of Googlebot to see if the site might have some firewall rules on that file that are blocking Google.
• Grep the logs to see if Googlebot has fetched the robots.txt file. That would tell you if they are getting a 200 response or something like a 4xx (client error) or 5xx (server error). If you don't see any hits from Google on robots.txt in the logs, then they might be getting blocked at another layer, like Cloudflare.

Outside of that, it seems to me that it's a server/DNS issue with your hosting company and will most likely work itself out (I know how that sounds, but it's true).

 

[Lifeh18]

Do you have any caching plugins enabled? Are you using Litespeed?

I'd ask them to double check the blocked IPs again.

Also, what does your robot.txt file look like? Scroll through it and see if it's disallowing Googlebot to crawl.

I did some extra digging around for you and found this:

A couple of places to search for clues:

• Use curl (or similar program) to fetch the robots.txt file with a user-agent of Googlebot to see if the site might have some firewall rules on that file that are blocking Google.
• Grep the logs to see if Googlebot has fetched the robots.txt file. That would tell you if they are getting a 200 response or something like a 4xx (client error) or 5xx (server error). If you don't see any hits from Google on robots.txt in the logs, then they might be getting blocked at another layer, like Cloudflare.

Outside of that, it seems to me that it's a server/DNS issue with your hosting company and will most likely work itself out (I know how that sounds, but it's true).

I will ask them again about the ips. i dont use any caching plugins

My robots.txt is: (I haven't changed anything here, so I don't believe it's this issue)

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

 

[Lifeh18]

My hosting says everything is fine on their side, google ips are already whitelisted. So it can't be my hosting issue

 

[hideath]

My hosting says everything is fine on their side, google ips are already whitelisted. So it can't be my hosting issue

are you using CF?
did you double check the htaccess file?
are you sure you're using the right web url base in the GSC (www, https, etc...)?

pm me the website url, i'll make a quick test

 

[lisanspace]

• Check Robots.txt File:
  • Ensure that the robots.txt file exists on your server and is accessible at http://yourdomain.com/robots.txt.
  • Verify the file's permissions to ensure it is readable by everyone (usually permissions set to 644).
• Server Configuration:
  • Check your server settings to ensure there are no restrictions or errors that might prevent the robots.txt file from being served.
  • Look into server logs to identify any potential issues or errors related to the robots.txt file.
• DNS Issues:
  • Ensure there are no DNS issues that might be preventing the tool from resolving your domain correctly.
• Firewall/Hosting Provider:
  • Sometimes firewalls or security settings on your hosting provider’s side might block access to certain files. Ensure your hosting provider is not blocking access to the robots.txt file.
• Temporary Glitches:
  • Occasionally, this issue might be due to a temporary glitch with your hosting provider or network. Try accessing the file after some time.
• Crawler/Indexing Tool:
  • Verify that the problem is not specific to the tool you're using. Try accessing the robots.txt file using a different tool or manually through a browser.

 

[Lifeh18]

are you using CF?
did you double check the htaccess file?
are you sure you're using the right web url base in the GSC (www, https, etc...)?

pm me the website url, i'll make a quick test

I'm not using Cloudfare. What do you look for in the httaccess?

Weird thing is, I never change anything and this problem started on its own.