Google low crawl rate

[OriginWorld]

Hi friends,

I got a website, which is hated by google. Most of my pages got deindexed and the crawl rate was reduced massively.

I thought maybe I could give my important pages more crawl budget by putting thousands of pages to disallow in robots.txt.
My website offers ebooks, which means one book can have hundreds of pages (chapters). I set robots.txt to disallow all chapter pages but the first 3 of every book.

However, my pages still don't get indexed anymore and the crawl rate is very low (6-17), while Bing and Yandex are crawling 300 pages each.

Does anyone have an idea how to solve this?

 

[Steptoe]

Are you actively building backlinks to your site? Crawl budget is mainly determined by popularity ('URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our index.') - meaning link authority, and overall health, meaning lack of error responses and connectivity issues.

If you are actively building backlinks, you could look at how your internal linking is structured. Aim to have every page ideally no more than 3 clicks away from your home page or main entry points, and have every page linked to from multiple other pages on the site.

I wouldn't recommend suddenly setting thousands of existing pages to disallow, as that could look like an error.

 

[OriginWorld]

Are you actively building backlinks to your site? Crawl budget is mainly determined by popularity ('URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our index.') - meaning link authority, and overall health, meaning lack of error responses and connectivity issues.

If you are actively building backlinks, you could look at how your internal linking is structured. Aim to have every page ideally no more than 3 clicks away from your home page or main entry points, and have every page linked to from multiple other pages on the site.

I wouldn't recommend suddenly setting thousands of existing pages to disallow, as that could look like an error.

Thanks for your reply! I created some good backlinks, but it didn't seem to help. And I'm afraid to buy backlinks because 1. My website is very niche, 2. the cost, 3. I don't want to risk google penalty

I did work on my website to improve the page weight, loading speed and internal linking, but I'm afraid that if the ebook pages and the 3 chapters per book don't get indexed then all the other thousands of pages also won't get indexed.

So you think I should allow the chapter pages again? If yes, should I allow them all at once or in batches?

 

[Aeroweb_Tech_Solutions]

Hi friends,

I got a website, which is hated by google. Most of my pages got deindexed and the crawl rate was reduced massively.

I thought maybe I could give my important pages more crawl budget by putting thousands of pages to disallow in robots.txt.
My website offers ebooks, which means one book can have hundreds of pages (chapters). I set robots.txt to disallow all chapter pages but the first 3 of every book.

However, my pages still don't get indexed anymore and the crawl rate is very low (6-17), while Bing and Yandex are crawling 300 pages each.

Does anyone have an idea how to solve this?

• Check Google Search Console for manual actions or penalties, and ensure your content is unique, valuable, and free from duplicate or spammy elements.
• Use <meta name="robots" content="noindex"> for low-value pages instead of disallowing them in robots.txt, and ensure your sitemap highlights important pages.
• Optimize your internal linking structure, improve server response times, and enhance page load speeds to boost crawl efficiency.
• Add unique content like summaries or reviews to enhance the value of your e-book pages and avoid being flagged as low-quality.
• Drive organic traffic and encourage user engagement to signal value to Google while consistently updating and maintaining your site.
• If no improvement is seen, consider a reconsideration request in Search Console and be patient, as recovery takes time.

 

[Nikolaos]

Hi friends,

I got a website, which is hated by google. Most of my pages got deindexed and the crawl rate was reduced massively.

I thought maybe I could give my important pages more crawl budget by putting thousands of pages to disallow in robots.txt.
My website offers ebooks, which means one book can have hundreds of pages (chapters). I set robots.txt to disallow all chapter pages but the first 3 of every book.

However, my pages still don't get indexed anymore and the crawl rate is very low (6-17), while Bing and Yandex are crawling 300 pages each.

Does anyone have an idea how to solve this?

How long did it take for you to build the site. If you scale too fast and put too much content (thousands of pages) google doesn't like it and might penalise you.

 

[OriginWorld]

How long did it take for you to build the site. If you scale too fast and put too much content (thousands of pages) google doesn't like it and might penalise you.

It took a few months to build the first MVP.

I put the website online in May 2024 and over the next months I developed the website heavily (content, functionality, SEO). There were times when I put 400 new pages per day to the website.

The only penalty I could get is hidden one, because everything is ok on GSC. I'm not doing cloaking or keywords stuffing and the domain is completely clean/new.

The weird thing: I have other websites, which also don't rank on google.
The only websites which get real impressions are an unfinished SAAS homepage and my dev website

Edit: I forgot to mention, that all websites that don't rank well are behind Cloudflare. However, the Googlebot can still access my pages without problem

 

[jyciap11]

It took a few months to build the first MVP.

I put the website online in May 2024 and over the next months I developed the website heavily (content, functionality, SEO). There were times when I put 400 new pages per day to the website.

The only penalty I could get is hidden one, because everything is ok on GSC. I'm not doing cloaking or keywords stuffing and the domain is completely clean/new.

The weird thing: I have other websites, which also don't rank on google.
The only websites which get real impressions are an unfinished SAAS homepage and my dev website

Edit: I forgot to mention, that all websites that don't rank well are behind Cloudflare. However, the Googlebot can still access my pages without problem

I created a website in September, 300 posts a day, all with AI. I had some traffic and, suddenly, Google deindexed almost the entire site. I asked several times via GSC for manual indexing but did not work. I then used an external indexer (you can find many here on BHW) to completely index my entire site and, after a couple of weeks, all 1000 pages were back on Google and so was the traffic. Now the site has been growing day after day for almost 3 months. all my sites are under cloudflare.
You can try, for a few bucks you can try to index again the pages.

 

[OriginWorld]

I created a website in September, 300 posts a day, all with AI. I had some traffic and, suddenly, Google deindexed almost the entire site. I asked several times via GSC for manual indexing but did not work. I then used an external indexer (you can find many here on BHW) to completely index my entire site and, after a couple of weeks, all 1000 pages were back on Google and so was the traffic. Now the site has been growing day after day for almost 3 months. all my sites are under cloudflare.
You can try, for a few bucks you can try to index again the pages.

Thanks for your recommendation! I'm trying out Linkdexing now, I hope it does something positive

 

[Aeroweb_Tech_Solutions]

Hi friends,

I got a website, which is hated by google. Most of my pages got deindexed and the crawl rate was reduced massively.

I thought maybe I could give my important pages more crawl budget by putting thousands of pages to disallow in robots.txt.
My website offers ebooks, which means one book can have hundreds of pages (chapters). I set robots.txt to disallow all chapter pages but the first 3 of every book.

However, my pages still don't get indexed anymore and the crawl rate is very low (6-17), while Bing and Yandex are crawling 300 pages each.

Does anyone have an idea how to solve this?

• Check Google Search Console for manual actions, security issues, and coverage errors.
• Use the URL Inspection Tool to analyze non-indexed pages.
• Verify robots.txt rules to avoid over-disallowing and ensure important pages are accessible.
• Confirm important pages do not have noindex meta tags or restrictive HTTP headers.
• Enhance internal linking to prioritize key pages.
• Secure high-quality backlinks to improve page importance.
• Analyze server logs to identify crawling patterns and issues with Googlebot.
• Submit a clean XML sitemap with priority pages in Google Search Console.
• Consolidate or enhance thin or low-quality content, such as sparse chapter pages.
• Address duplicate content by using canonical tags for main pages.
• Optimize the site for crawl efficiency by reducing unnecessary resources and improving speed.
• Focus on user engagement signals like reduced bounce rates and mobile usability.

 

[wuuwuw]

W

I created a website in September, 300 posts a day, all with AI. I had some traffic and, suddenly, Google deindexed almost the entire site. I asked several times via GSC for manual indexing but did not work. I then used an external indexer (you can find many here on BHW) to completely index my entire site and, after a couple of weeks, all 1000 pages were back on Google and so was the traffic. Now the site has been growing day after day for almost 3 months. all my sites are under cloudflare.
You can try, for a few bucks you can try to index again the pages.

What external indexer did you use?

 

[OriginWorld]

Update:
I removed the robots.txt block.
I have used Linkdexing and an other indexer for a few high quality pages.
Result:
Google increased my crawl rate, but the pages are still not indexed.
No matter what I do, I can't get google to index pages. There is no message from Google's side. No error message nor hint.

 

[jyciap11]

Aggiornamento:
Ho rimosso il blocco robots.txt.
Ho utilizzato Linkdexing e un altro indicizzatore per alcune pagine di alta qualità.
Risultato:
Google ha aumentato la velocità di scansione, ma le pagine non sono ancora indicizzate.
Non importa cosa faccia, non riesco a far indicizzare le pagine a Google. Non c'è alcun messaggio da parte di Google. Nessun messaggio di errore né suggerimento.

Any update?

 

[OriginWorld]

Any update?

Turned bad again. Afterwards I found an error in my cloudflare WAF config, which blocked a few googleBots, however it didn't fix the crawl rate.

The big chang happened 2 - 4 weeks ago, when google randomly started crawling hundreds of my pages and indexed them right away. I don't know if this is a lasting effect or if google will once again show me its cold shoulder.

 

[walangalam2]

Turned bad again. Afterwards I found an error in my cloudflare WAF config, which blocked a few googleBots, however it didn't fix the crawl rate.

The big chang happened 2 - 4 weeks ago, when google randomly started crawling hundreds of my pages and indexed them right away. I don't know if this is a lasting effect or if google will once again show me its cold shoulder.

I think this was after the google core update

 

[Breaking Vibes]

Disallowing pages in robots.txt doesn’t increase crawl budget it blocks them entirely, so focus on improving content quality and internal linking to boost crawl and indexing

 

[OriginWorld]

I think this was after the google core update

Might be.
But it reverted again. Low crawl rate and hundreds of pages got unindexed.

 

[Ven_doora]

If Google is deindexing your pages and drastically reducing your crawl rate, it’s a strong signal that it perceives your site as low-quality, thin, or potentially violating guidelines — especially common with sites offering ebooks or multi-page content.
Don’t block via robots.txt — use noindex, follow instead.
Focus on unique, useful content and clean up thin or duplicated pages.
Improve internal linking and boost external interest via backlinks.
Submit only the best URLs in sitemap and GSC.

 

[OriginWorld]

Hello friends,

I might have found the issue, which made Google hate me. I didn't know that Google uses HEAD requests for crawling. Since my website is made from scratch, I didn't support HEAD requests, which gave Google 404 errors. Unfortunately, Google doesn't indicate if there are problems with your HEAD requests.

So the past year, Google always got 404s and made it think that my website is trashy.

I enable the HEAD requests now and got more traffic in the next day. However, it's not a guarantee if this is really the problem. I need more time to see the long time results.

 

[2165982]

I think you should check Google Search Console and see if there are any issues with your website that Google is reporting. It is possible that there are other reasons why your pages are not getting indexed apart from robots.txt. It may be helpful to also submit a sitemap to Google to help them crawl your website more efficiently. Additionally, you can try to improve the quality of your content and ensure that your website is following Google's webmaster guidelines. Good luck!

 

[Jenny565]

I think this was after the google core update

So what should we do when we encounter this situation to maintain and develop the site?