1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Anyone Google crawler lag? No index for new articles?

Discussion in 'Black Hat SEO' started by KHer0, Feb 18, 2017.

  1. KHer0

    KHer0 Supreme Member

    Joined:
    Mar 22, 2011
    Messages:
    1,228
    Likes Received:
    1,155
    Occupation:
    Architect
    For the last 2 months I have been experiencing pretty weird crawler indexation for all of my websites.

    I know how to write content that ranks and how to create quality websites, but it's the first time I have seen this.

    The crawler visits the fresh website and indexes all articles on it. Cuz of my keyword research and Content all my keywords appear in top 30 and harder ones in top 50. So, this should be a good sign for google it's a good website and they should visit it more, No, the crawler disappears for weeks and doesn't index any changes to old posts nor index new posts.

    First I have tried this one mini-niche website. It took google one week to index homepage and another week to index the rest of the website (4 pages) and the most recent articles haven't been indexed till now (4 weeks!)

    I made a try with another domain but this time I dripped feed 2-3articles/day so that the crawler loves it and come everyday looking for the new piece of content.

    Within a week google indexed the 11 pages and put them in Amazing positions. I have traffic already, and many keywords in page 2 while the hardest ones in page 4. So, looks like great content that google loved it. I continued the drip feed of articles for a whole week and google neither indexed them, nor visited my website !!

    I know I can index it by backlinks, or waiting .. But is this how the crawler is working now? Taking weeks to index new articles ! Is google definition of update means making it slower? Or am I the only seeing this?
     
  2. KHer0

    KHer0 Supreme Member

    Joined:
    Mar 22, 2011
    Messages:
    1,228
    Likes Received:
    1,155
    Occupation:
    Architect
    Really, No one has anything to say about this?
     
  3. vancleve

    vancleve Jr. VIP Jr. VIP

    Joined:
    Apr 4, 2014
    Messages:
    119
    Likes Received:
    20
    Just stumbled upon this thread now and yup, I'm seeing the same thing. I also started a blog on an expired domain. It looks like Google visits my website every week or so, which means a longer waiting period to index new articles or refresh old ones after updating them.
     
  4. hay706

    hay706 Junior Member

    Joined:
    Mar 6, 2016
    Messages:
    119
    Likes Received:
    38
  5. yugejackpots

    yugejackpots BANNED BANNED

    Joined:
    Apr 16, 2016
    Messages:
    193
    Likes Received:
    77
    Definitely seems like there's multiple "tiers" to the way they're indexing now. They'll index the homepage, let it chill, come back, index a few more articles, chill, then a few weeks later index the rest of the site. I assume the idea is to make sure blackhat's don't instantly add links to domains, that's my theory at least. I wait a few weeks before adding OBLs to fresh link domains nowadays.
     
  6. darulez

    darulez Jr. VIP Jr. VIP

    Joined:
    Mar 12, 2013
    Messages:
    2,256
    Likes Received:
    700
    Gender:
    Female
    Occupation:
    Waiting 36 days till I can stick it in
    Location:
    Walhalla
    nope. but I submit manually each new page...
     
  7. 710fla

    710fla Jr. VIP Jr. VIP

    Joined:
    Aug 25, 2015
    Messages:
    650
    Likes Received:
    172
    Setup IFTTT recipe to auto post your new website page links on Twitter, Facebook, Tumblr, etc.

    Google will index all new pages quickly.
     
  8. KHer0

    KHer0 Supreme Member

    Joined:
    Mar 22, 2011
    Messages:
    1,228
    Likes Received:
    1,155
    Occupation:
    Architect
    Sorry guys, I didnt get any notification till darulez quoted me

    Here is the update, I ended up using fetch url and submit to index in webmaster tools. There result was pretty strange, my article got indexed in seconds and in great spots. I repeated the process many time to make sure

    So, I believe the crawler is pretty fast at indexing, pretty slow at crawling. It's about making the crawler decide to index your page, but the algoritm to define where it should rank takes seconds.

    And it took exactly 3.5 weeks for the website to mature and see your interlinking and content optimization effect. The website is 4 weeks now and showing up for more and more keywords every single day without any backlinks or adding new content. I shall start my off page work this month

    Hope this helps
     
    • Thanks Thanks x 1
  9. vancleve

    vancleve Jr. VIP Jr. VIP

    Joined:
    Apr 4, 2014
    Messages:
    119
    Likes Received:
    20
    I did the same. The strange thing was not all fetched URLs got indexed. Some pages took longer before showing up on the SERP. And for websites not linked to GWT, it seems I have no other choice but to wait.
     
  10. KHer0

    KHer0 Supreme Member

    Joined:
    Mar 22, 2011
    Messages:
    1,228
    Likes Received:
    1,155
    Occupation:
    Architect
    Here is a great tip. Don't Ever show same text on two pages on ur website even if it was just the title.

    So, ur seo title should be different from slug and different from h1 title.

    If you are using normal categories then no index them with author, tags, taxonomies

    Add meta description and excerpt to all articles

    To conclude, Each single post or page on ur website should have completely unique text
     
    • Thanks Thanks x 1