1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Barney update is all about the indexer Caffeine

Discussion in 'White Hat SEO' started by bartosimpsonio, Apr 6, 2017.

  1. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,485
    Likes Received:
    11,186
    Occupation:
    CHEAP
    Location:
    DATASETS
    Home Page:
    That Google person on Twitter said they have separated even further the crawler from the indexer.

    They said the indexer requests the crawler to do work. I always thought it was the opposite, that the crawler asked the indexer to do work. Not so, if we take his word at face value.

    I'm inclined to believe him because this makes sense right now.

    The indexer has pages in its index (duh). It's a big ass database. That database has links out with tags like do or no follow. The ones with do follow are sent from the indexer to the crawler for followup crawling. Which means the indexer is driving the crawler.

    When a site is penalized, this penalty is done at the indexer. So the crawler does not go to the penalized site. Makes sense, why keep wasting work on pages that'll never show?

    What has happened after "Fred" or Barney or whatever you wanna call it?

    The indexer got tougher to get into. The indexer gets hardened, it requests less crawls from the crawler.

    It's getting harder to get into Caffeine's brain, and if the urls aren't in Caffeine, it'll never get crawled. Also something else: the index is becoming stale with urls in it, because the crawler is no longer commanding what's new, the index determines what gets in, so this explains why it takes ages to get a nice complete crawl. First you need to get discovered AND indexed. FROM the index, it commands the crawler to bring in more pages from your site.

    This makes the "population" of the index older...Thus you keep getting crawls for ancient URLs....some urls that don't exist. I administer sites from the 90's, I get Googlebot requests for URLs that were Clinton era....

    So to control who gets into the index, they made the indexer harder. And also older and more stale.

    I think that's what this Fred update was about : hardening the indexer called Caffeine.

    Indexing is taking longer, and the indexer commands the crawler. This way pages don't even reach the index for weeks, months. Before that, the crawler commanded the indexer : hey I got new material, index it! Now it's the opposite.

    Just my 2 cents about this Wilma update.
     
    • Thanks Thanks x 2
  2. jstover77

    jstover77 Executive VIP Jr. VIP

    Joined:
    Feb 27, 2009
    Messages:
    2,376
    Likes Received:
    3,621
    Gender:
    Male
    Occupation:
    Entrepreneur
    Location:
    Doylestown PA
    Home Page:
    I love his Twitter feed. So many knowledge bombs being dropped.

    Good stuff bro!
     
    • Thanks Thanks x 1
  3. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,485
    Likes Received:
    11,186
    Occupation:
    CHEAP
    Location:
    DATASETS
    Home Page:
    Seems like a more likeable guy than the one before him. More legit at least IMO.

    But watch out for some dubious stuff he pulls out of nowhere. Recently he said something about Google using DMOZ snippets.....DMOZ was already closed in March.
     
    • Thanks Thanks x 1
  4. jjhost12

    jjhost12 Jr. VIP Jr. VIP

    Joined:
    Aug 11, 2015
    Messages:
    117
    Likes Received:
    23
    Thanks for the thread, methode seems like he knows his shit lots of twitter posts regarding SEO. Nice find! :)
     
    • Thanks Thanks x 1
  5. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,485
    Likes Received:
    11,186
    Occupation:
    CHEAP
    Location:
    DATASETS
    Home Page:
    He's the new Matt Cutts - that's straight from Google's headquarters info. Sure knows a lot ;)
     
  6. scnonah

    scnonah Jr. VIP Jr. VIP

    Joined:
    Jan 2, 2017
    Messages:
    224
    Likes Received:
    124
    Gender:
    Male
    Hmm, I can't believe anything he says. IMO it's disinformation.
     
  7. jstover77

    jstover77 Executive VIP Jr. VIP

    Joined:
    Feb 27, 2009
    Messages:
    2,376
    Likes Received:
    3,621
    Gender:
    Male
    Occupation:
    Entrepreneur
    Location:
    Doylestown PA
    Home Page:
    Yeah I read that. Obviously, a good understanding of SEO should be in place when reading anything in this industry. Lol..
     
    • Thanks Thanks x 1