Technical Issue Regarding Google Bot - Any Insights?

Danny1234

Registered Member
Joined
Jun 22, 2017
Messages
51
Reaction score
10
Looking to see if anyone here has run into a similar technical issue and whether or not I may be overthinking this or not. Judging by the amount of errors, I'm not sure I am.

Frustratingly, my traffic has fallen off of a cliff over the last while and notable improvements to content etc haven't really caused an increase. Underlying engagement metrics are strong, and realistically my mind is absolutely blown as to how this can be happening.

I had someone from an SEO group I'm in ask to dig into our server logs, and they found a huge chunk of 504 errors directly correlated to google bot traffic. Nearly 1/3 of all googlebot requests hit a 504 error. Out of 1115 Googlebot records today, 300~ of them are 504 errors.

When I look to the type of content, it is all content pages.

In contrast to this, virtually zero visitors to the website encountered this error.

I contacted my host, who has no idea where the excess 504 errors would be coming from on the bot side of things. They started a ticket, but I don't expect to get too much progress on that side of things.

Has anyone else had this sort of technical error, and if so, what have you done to fix it?
 
Apparently it's a host's problem. The reverse proxy is configured on your server and it throws error 504 when this proxy doesn't get a response from your website in timely manner. It can also happen if your website is behind CloudFlare. In any case your host doesn't serve your website well.
 
This is related to how your server handles bot traffic or just the volume of requests coming in. Talk to your host again and ask if there are any rate-limiting rules or firewalls that might be blocking or slowing down Googlebot. You could also try adjusting crawl settings in GSC to limit how often the bot hits your site.
 
Back
Top