Anyone Google crawler lag? No index for new articles?

For the last 2 months I have been experiencing pretty weird crawler indexation for all of my websites.

I know how to write content that ranks and how to create quality websites, but it's the first time I have seen this.

The crawler visits the fresh website and indexes all articles on it. Cuz of my keyword research and Content all my keywords appear in top 30 and harder ones in top 50. So, this should be a good sign for google it's a good website and they should visit it more, No, the crawler disappears for weeks and doesn't index any changes to old posts nor index new posts.

First I have tried this one mini-niche website. It took google one week to index homepage and another week to index the rest of the website (4 pages) and the most recent articles haven't been indexed till now (4 weeks!)

I made a try with another domain but this time I dripped feed 2-3articles/day so that the crawler loves it and come everyday looking for the new piece of content.

Within a week google indexed the 11 pages and put them in Amazing positions. I have traffic already, and many keywords in page 2 while the hardest ones in page 4. So, looks like great content that google loved it. I continued the drip feed of articles for a whole week and google neither indexed them, nor visited my website !!

I know I can index it by backlinks, or waiting .. But is this how the crawler is working now? Taking weeks to index new articles ! Is google definition of update means making it slower? Or am I the only seeing this?

Really, No one has anything to say about this?

Just stumbled upon this thread now and yup, I'm seeing the same thing. I also started a blog on an expired domain. It looks like Google visits my website every week or so, which means a longer waiting period to index new articles or refresh old ones after updating them.

You can get indexed instantly if you submit here: https://www.google.com/webmasters/tools/submit-url

Don't use it for a pbn though.

Definitely seems like there's multiple "tiers" to the way they're indexing now. They'll index the homepage, let it chill, come back, index a few more articles, chill, then a few weeks later index the rest of the site. I assume the idea is to make sure blackhat's don't instantly add links to domains, that's my theory at least. I wait a few weeks before adding OBLs to fresh link domains nowadays.

KHer0 said:

Really, No one has anything to say about this?

Click to expand...

nope. but I submit manually each new page...

Setup IFTTT recipe to auto post your new website page links on Twitter, Facebook, Tumblr, etc.

Google will index all new pages quickly.

Sorry guys, I didnt get any notification till darulez quoted me

Here is the update, I ended up using fetch url and submit to index in webmaster tools. There result was pretty strange, my article got indexed in seconds and in great spots. I repeated the process many time to make sure

So, I believe the crawler is pretty fast at indexing, pretty slow at crawling. It's about making the crawler decide to index your page, but the algoritm to define where it should rank takes seconds.

And it took exactly 3.5 weeks for the website to mature and see your interlinking and content optimization effect. The website is 4 weeks now and showing up for more and more keywords every single day without any backlinks or adding new content. I shall start my off page work this month

Hope this helps

KHer0 said:

Here is the update, I ended up using fetch url and submit to index in webmaster tools. There result was pretty strange, my article got indexed in seconds and in great spots. I repeated the process many time to make sure

Click to expand...

I did the same. The strange thing was not all fetched URLs got indexed. Some pages took longer before showing up on the SERP. And for websites not linked to GWT, it seems I have no other choice but to wait.

vancleve said:

I did the same. The strange thing was not all fetched URLs got indexed. Some pages took longer before showing up on the SERP. And for websites not linked to GWT, it seems I have no other choice but to wait.

Click to expand...

Here is a great tip. Don't Ever show same text on two pages on ur website even if it was just the title.

So, ur seo title should be different from slug and different from h1 title.

If you are using normal categories then no index them with author, tags, taxonomies

Add meta description and excerpt to all articles

To conclude, Each single post or page on ur website should have completely unique text