Analyzing server logs for Google Bot crawl budget and AI bot checks

Server logs are still the only place you can see what bots actually do on your site. GSC gives partial metrics and most AI platforms report almost nothing useful.

Oncrawl dropped an e-book on four log file use cases that walks through turning raw hits into something you can act on.

One part that stands out is catching intermittent server errors and empty pages that still return 200 before they cost traffic. Another is seeing where Google and AI systems actually spend crawl budget page group by page group, plus the levers that move it.

They also cover verifying AI bot behavior. That means checking whether GPTBot is really GPTBot, separating training crawlers from user fetchers, and measuring real AI search click-through. Last use case fills the holes that ad blockers and consent leave when analytics sources disagree.

Takes from Lily Ray, Detlef Johnson and a few others are in there. I would pull the last 30 days of my own logs next and map bot hits against GSC crawl stats to spot the gaps.

Hope some of you find it useful.

Too much trouble for my small sites. The last time I analysed logs was for a client using the ELK stack, and I loved it but too lazy to do it again.

The empty pages returning 200 caught my eye. Soft 404s like that hide from status codes, and a sudden drop in response size in the logs can flag them if the size is recorded. I find that more useful day to day than counting Googlebot visits.