Which criteria use Google to index posts?

J

jyciap11

Elite Member
Joined
Apr 11, 2014
Messages
1,650
Reaction score
441
Please, don’t tell me “HQ posts, good internal links etc.” This things not working, there are no Rules.

I have a couple of websites with 1000 IDENTICAL articles, only a few words change but they are identical. I have 1000 products with the same description, only changed the product name. I continue to publish about 2 posts per day. Some get indexed instantly, some later, some NEVER.
What is different between one post and another? Why are some indexed right away and others not? The posts are identically, so the HQ posts, internal links etc. doesn’t the cause, Because all posts are identically.
I have a new site with 20 UNIQUE articles and none have been indexed for 20 days, even though I submitted the posts to GSC.
This makes no sense.
so my question is this: if i have 1000 identical posts, where i changed the product name, why does google index some of them, while others are "crawled but not indexed"?
since they are identical they should all be indexed, or none, or only one and the others not due to duplicate content, am I right?
 
Advertise on BHW
There could be several reasons why Google might not be indexing your new website or its articles:

Technical issues: Check for errors in your robots.txt file that might be blocking crawling. Also, ensure that your "noindex" meta tags are set up correctly on your pages.

Lack of backlinks: New sites without backlinks may receive less attention from Google. If your site doesn't have external links, it may be harder for it to get indexed.

Slow page loading: If your website loads slowly, it could affect Google's ability to crawl it. Make sure your site is optimized for speed.

Low-quality content: If your content is deemed low-quality, duplicated, or uninformative, Google may decide not to index it. Uniqueness and value of content are crucial for indexing.

Hosting issues: If your site is frequently down or has technical glitches, it can impact indexing.

Incorrect URL structure: Ensure that your URLs are correctly structured and free of errors.

Use of a new domain: If you're using a new domain, Google may need more time to index it. Sometimes, new domains require additional time to be noticed.
 
OracleBo said:
There could be several reasons why Google might not be indexing your new website or its articles:

Technical issues: Check for errors in your robots.txt file that might be blocking crawling. Also, ensure that your "noindex" meta tags are set up correctly on your pages.

Lack of backlinks: New sites without backlinks may receive less attention from Google. If your site doesn't have external links, it may be harder for it to get indexed.

Slow page loading: If your website loads slowly, it could affect Google's ability to crawl it. Make sure your site is optimized for speed.

Low-quality content: If your content is deemed low-quality, duplicated, or uninformative, Google may decide not to index it. Uniqueness and value of content are crucial for indexing.

Hosting issues: If your site is frequently down or has technical glitches, it can impact indexing.

Incorrect URL structure: Ensure that your URLs are correctly structured and free of errors.

Use of a new domain: If you're using a new domain, Google may need more time to index it. Sometimes, new domains require additional time to be noticed.
Click to expand...
Please, dont use ChatGPT to write this (useless) posts.
I already told that the content is the same for all posts, I changed only the product name, so your answers don’t have sense.
 
google oftentimes makes no sense... and I think they've even given up on their search engine, their main focus is to make more money (whichever way they can) before they die out, so don't act surprised when you see inconsistencies...

Also, google doesn't like duplicate content in general and they usually don't index it, and the fact that some of your duplicate pages got indexed is actually pretty lucky.

Internal links help with crawlability (which I'm sure you appreciate when you have 1000s of pages), user experience, and boosting up your site's authority and topicality, so I'd say they're pretty important despite of you being pissed off about them :)
 
tazarbm said:
google oftentimes makes no sense... and I think they've even given up on their search engine, their main focus is to make more money (whichever way they can) before they die out, so don't act surprised when you see inconsistencies...

Also, google doesn't like duplicate content in general and they usually don't index it, and the fact that some of your duplicate pages got indexed is actually pretty lucky.

Internal links help with crawlability (which I'm sure you appreciate when you have 1000s of pages), user experience, and boosting up your site's authority and topicality, so I'd say they're pretty important despite of you being pissed off about them :)
Click to expand...
it's not luck, I've been doing this for ten years. before I got a manual penalty, one of my sites had 4000 posts, all identical except for the product name. 2.5k pages were indexed from 2016 to 2024.
luck?
I've had dozens of websites like this in 15 years of experience, never had any problems.
but I've always wondered why you index some pages and not others, it doesn't make sense.
I've made two more identical ones in the last few months, some pages are indexed, others are not.
I repeat, they are identical, only the product name changes.
 
jyciap11 said:
it's not luck, I've been doing this for ten years. before I got a manual penalty, one of my sites had 4000 posts, all identical except for the product name. 2.5k pages were indexed from 2016 to 2024.
luck?
I've had dozens of websites like this in 15 years of experience, never had any problems.
but I've always wondered why you index some pages and not others, it doesn't make sense.
I've made two more identical ones in the last few months, some pages are indexed, others are not.
I repeat, they are identical, only the product name changes.
Click to expand...
you've gotten away with it for too long if you asked me :)

Anyway, it could be the crawl budget then, or a combination of crawl budget + duplicate content. If google sees that 1000 pages are identical and the crawl budget they've allocated to your site is low they have no reasons to waste it on crawling and indexing the copycats. Just my $0.02....
 
tazarbm said:
you've gotten away with it for too long if you asked me :)

Anyway, it could be the crawl budget then, or a combination of crawl budget + duplicate content. If google sees that 1000 pages are identical and the crawl budget they've allocated to your site is low they have no reasons to waste it on crawling and indexing the copycats. Just my $0.02....
Click to expand...
And how can I increase this budget?
 
jyciap11 said:
And how can I increase this budget?
Click to expand...
you can't increase it, only google can. But you can optimize it by following a few steps, as outlined here:
https://www.searchenginejournal.com/technical-seo/crawl-budget/
https://backlinko.com/hub/seo/crawl-budget

But in all seriousness, your problem is most likely the duplicate content, that's what I would focus on. Just because you've run away with it for 10 years doesn't mean that you can continue doing so.

Just do a test and replace the duplicate content with 100% unique and high quality manual content on 10-20 pages that have been crawled but not indexed and then ask for indexation of these pages manually in your GSC. If google does index them you will know that duplication / low quality content is the issue, if they still won't index.... well... in this case there might be something wrong with your site, or even with the domain itself. I've heard some folks say that google does blacklist / flag domain names that they consider spammy or low quality, so if you've been marked it's no surprise that google refuses to deal with your site.
 
tazarbm said:
you can't increase it, only google can. But you can optimize it by following a few steps, as outlined here:
https://www.searchenginejournal.com/technical-seo/crawl-budget/
https://backlinko.com/hub/seo/crawl-budget

But in all seriousness, your problem is most likely the duplicate content, that's what I would focus on. Just because you've run away with it for 10 years doesn't mean that you can continue doing so.

Just do a test and replace the duplicate content with 100% unique and high quality manual content on 10-20 pages that have been crawled but not indexed and then ask for indexation of these pages manually in your GSC. If google does index them you will know that duplication / low quality content is the issue, if they still won't index.... well... in this case there might be something wrong with your site, or even with the domain itself. I've heard some folks say that google does blacklist / flag domain names that they consider spammy or low quality, so if you've been marked it's no surprise that google refuses to deal with your site.
Click to expand...
I understand your point, but I'm not sure about this.
I tried to add one new posts (same posts as the others, I only changed the product name as always) in one website this morning and sent to GSC, got indexed in about 2 hours (domain was created on 2016, always mine).
I don't think it depends on the content, which even if almost identical to the rest is still high quality, I think it depends on how "old" the domain is.
I have some sites made a couple of months ago with original content that are struggling to get indexed, I think it all depends on how old the domain is.
 
The main reason some of your posts are indexed while others aren't could be due to Google's treatment of duplicate content. Even though you're changing only the product name, the posts are still largely identical. Google tends to prioritize unique, valuable content, and when it detects too much similarity, it may choose not to index certain posts because they don't provide enough unique information. Also, factors like site structure, crawl budget, and even server issues can impact which pages get indexed. The "crawled but not indexed" status means Google saw the page but didn't find enough value in it to index right away.
 
Sometimes they’ll index a few and ignore the rest because they think there’s no value in indexing all of them.
 
jyciap11 said:
I think it depends on how "old" the domain is.
Click to expand...
jyciap11 said:
I think it all depends on how old the domain is.
Click to expand...
you don't think that, you want to think that. Also, domain age is not important (unless it's a new, last minute thing that google have implemented that nobody knows about yet)...

What is important are domain history and cleanness, backlink profile, and content history and quality are important....

Anyway, if the copycats and low quality content are not the issue I don't know what else could be. From experience, I know that these are some of the factors that impact indexation the most but if you know otherwise no problem.

Good luck fixing it...
 
tazarbm said:
you don't think that, you want to think that. Also, domain age is not important (unless it's a new, last minute thing that google have implemented that nobody knows about yet)...

What is important are domain history and cleanness, backlink profile, and content history and quality are important....

Anyway, if the copycats and low quality content are not the issue I don't know what else could be. From experience, I know that these are some of the factors that impact indexation the most but if you know otherwise no problem.

Good luck fixing it...
Click to expand...
I agree with what you say, I'm just wondering why some posts are indexed and others are not? What's the logic behind it? They are duplicate content ok, then don't index any of them, right? Because many are not only indexed, but they are also on the first page.
no need to fix anything, probably the trick is to create 10,000 posts, home they will index 5,000 and that's it
 
jyciap11 said:
I'm just wondering why some posts are indexed and others are not? What's the logic behind it?
Click to expand...
I told you earlier that not everything at google makes sense, and I was speaking the truth.

The Google algo has over 2 billion of lines of code and - in order for Google to keep the secret over how their search engine works - they've split the code into multiple chunks and they've given each chunk of the code to different departments to deal with updates, fixes, etc...

So, as you can imagine, not a single google engineer has the full idea of how the algo works, every engineer knows only what they've given to work with. And when your right hand doesn't know what the left hand does, and neither hand doesn't know what each eye sees, but you ask them all to coordinate with each other... well, you can see why inconsistencies, bugs (and illogical stuff in general) happen.

The google algo is like a woman: you don't have to understand her, just work with her for better communion. I can't come up with a better analogy, but this one's pretty close to what's going on with google with regards to your question. Don't try to understand it, just do what you know works and if it doesn't work, shrug it off and move on...
 
tazarbm said:
I told you earlier that not everything at google makes sense, and I was speaking the truth.

The Google algo has over 2 billion of lines of code and - in order for Google to keep the secret over how their search engine works - they've split the code into multiple chunks and they've given each chunk of the code to different departments to deal with updates, fixes, etc...

So, as you can imagine, not a single google engineer has the full idea of how the algo works, every engineer knows only what they've given to work with. And when your right hand doesn't know what the left hand does, and neither hand doesn't know what each eye sees, but you ask them all to coordinate with each other... well, you can see why inconsistencies, bugs (and illogical stuff in general) happen.

The google algo is like a woman: you don't have to understand her, just work with her for better communion. I can't come up with a better analogy, but this one's pretty close to what's going on with google with regards to your question. Don't try to understand it, just do what you know works and if it doesn't work, shrug it off and move on...
Click to expand...
Yes, the only problem is that i dont know what to do, because some identical sites work well, they dont index all the content but they work. other unique and HQ dont work at all. the problem is always that stupid "crawled, not indexed".
i think the best thing is to have 100 websites, divided in groups of 20 made differently.
 
Google skips indexing some posts due to duplicate content, seeing them as low-value or repetitive.
 
No one knows exactly. But if content is good and there are high quality backlinks, it will eventually rank up. There are also some secrets to index faster like linking from some specific social media websites.
 
Google will choose to index only a small portion of your content and simply treat the rest as duplicate content.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top