Hi friends,
I got a website, which is hated by google. Most of my pages got deindexed and the crawl rate was reduced massively.
I thought maybe I could give my important pages more crawl budget by putting thousands of pages to disallow in robots.txt.
My website offers ebooks, which means one book can have hundreds of pages (chapters). I set robots.txt to disallow all chapter pages but the first 3 of every book.
However, my pages still don't get indexed anymore and the crawl rate is very low (6-17), while Bing and Yandex are crawling 300 pages each.
Does anyone have an idea how to solve this?