That Google person on Twitter said they have separated even further the crawler from the indexer. They said the indexer requests the crawler to do work. I always thought it was the opposite, that the crawler asked the indexer to do work. Not so, if we take his word at face value. I'm inclined to believe him because this makes sense right now. The indexer has pages in its index (duh). It's a big ass database. That database has links out with tags like do or no follow. The ones with do follow are sent from the indexer to the crawler for followup crawling. Which means the indexer is driving the crawler. When a site is penalized, this penalty is done at the indexer. So the crawler does not go to the penalized site. Makes sense, why keep wasting work on pages that'll never show? What has happened after "Fred" or Barney or whatever you wanna call it? The indexer got tougher to get into. The indexer gets hardened, it requests less crawls from the crawler. It's getting harder to get into Caffeine's brain, and if the urls aren't in Caffeine, it'll never get crawled. Also something else: the index is becoming stale with urls in it, because the crawler is no longer commanding what's new, the index determines what gets in, so this explains why it takes ages to get a nice complete crawl. First you need to get discovered AND indexed. FROM the index, it commands the crawler to bring in more pages from your site. This makes the "population" of the index older...Thus you keep getting crawls for ancient URLs....some urls that don't exist. I administer sites from the 90's, I get Googlebot requests for URLs that were Clinton era.... So to control who gets into the index, they made the indexer harder. And also older and more stale. I think that's what this Fred update was about : hardening the indexer called Caffeine. Indexing is taking longer, and the indexer commands the crawler. This way pages don't even reach the index for weeks, months. Before that, the crawler commanded the indexer : hey I got new material, index it! Now it's the opposite. Just my 2 cents about this Wilma update.