Robots.txt cannot be accessed

hpno

Newbie
Joined
Oct 29, 2023
Messages
14
Reaction score
1
I haven't changed anything in robots.txt file. But suddenly, GSC cannot access my sitemap anymore. The error is: general http error. When I use URL Inspection tool for checking the sitemap, the error is: Robots.txt cannot be accessed

However, the robots file is accessible. Tried to download it from multiple locations. Even pretended as Googlebots, I could access both sitemap and robot files.

Any idea what's happening and how to fix?
Thank you.
 
Found the reason. A plugin was working weirdly to make firewall blocking some IPs, unfortunately including some Google bots IP. Removing those IPs, then I could validate robots file again. And hopefully Google will validate the sitemap soon.
 
Found the reason. A plugin was working weirdly to make firewall blocking some IPs, unfortunately including some Google bots IP. Removing those IPs, then I could validate robots file again. And hopefully Google will validate the sitemap soon.
What plugin was the problem? So if someone face this problem in the future, it will be easier to find a quick solution
 
Back
Top