What's your workflow for identifying crawl budget waste on large site?

When am working on large site, crawl budget becomes key concern. And what the workflow for identifying crawl budget waste? Which tool or method that given you most reliable insights?

> I usually start with Google Search Console and server logs to spot pages that are being crawled but add little value. Cleaning up duplicate, thin, and unnecessary URLs has made the biggest difference in improving crawl efficiency.

I usually start with Google Search Console to see which pages are being crawled and indexed. Then I run a full site crawl to find duplicate pages, redirect chains, broken links, parameter URLs, and low value pages that should not be taking up crawl resources.

How large is large, though? Google's own crawl budget guide puts it at around a million pages, or ten thousand plus if the content changes daily. Below that it usually is not the bottleneck, and that is worth ruling out before you spend a week on it.

If you are genuinely in that range, both answers above will surface duplicates, but I would open the Crawl Stats report first and look at the breakdown by response code and by file type. A large share of requests coming back 404 or 301, or a lot of them going to JS and CSS, tells you something a site crawl cannot, because a crawler shows you what exists and the logs show you where Googlebot actually chose to spend the requests.

The one that gets missed most is soft 404s. Google calls them out specifically because they keep getting crawled, and they do not show up as errors in a normal crawl.

Server logs are still my starting point because they show what bots actually spend time on. After that I compare crawl data with index coverage and internal linking to uncover pages that consume budget without adding value.

I usually start with Google Search Console and server logs to see where Googlebot is spending its crawl budget. Then I look for duplicate URLs, parameter pages, orphan pages, and low-value content that can be consolidated or blocked from crawling.

I overlay Googlebot request paths with GSC performance to spot where the bot is burning requests on URLs that yield zero traffic.