I struggle how to index Programmatic SEO website correclty

S

sensei228

Newbie
Joined
Apr 22, 2021
Messages
4
Reaction score
0
Hello everyone.
Just for context, I'm a beginner in webdev and SEO, and this is my first project live.

It is programmatic SEO website based on comparisons (Product A vs Product B).

A few days ago I reached my first small milestone: my site started getting indexed.
But what's concerning is that Google started indexing not the pages I was expecting... (obviously my fault.)


How I'm handling indexing right now:
  • Added most content-rich comparison pages to sitemap (fewer than 100).
  • Each comparison page includes suggested comparisons, based on Product A and Product B. Suggestions are each time generated randomly. They are do-follow.
Currently total number of possible pairs is around 2 million pages.
And I'm afraid to waste all crawling budget for pages that won't drive any traffic.

To be honest, I think 99% of these comparisons wouldn't even make sense for people to search.
Most interest is likely in most popular products.
I believe the most of traffic potential lies in comparisons between top 1% pages from A and the top 1% pages from B.


A possible solution I'm considering is adding noindex to all of non-top pairs.


It's hard to say exactly what my question is - I'm just pretty confused rn.
I'd appreciate any advice, and would love to know if my concerns make sense, or where am I wrong.
 
Advertise on BHW
I’d noindex the low-value stuff and maybe even block some of it in robots. Let Google focus on what actually has a shot at ranking. Keep sitemap tight, prune often. That’s normal in prog SEO builds.
 
NeedMyLink said:
I’d noindex the low-value stuff and maybe even block some of it in robots. Let Google focus on what actually has a shot at ranking. Keep sitemap tight, prune often. That’s normal in prog SEO builds.
Click to expand...
Thanks for the advice!

Here's what I've implemented:
  • Set X-Robots-Tag = 'noindex, nofollow' for comparisons that aren't in top1%.
  • Suggested comparisons now link only to other top1% comparisons.
  • Generated and submitted a sitemap containing only top1% pages.
  • The rest of comparisons are still accessible via manual filtering, and suggestion links on such pages are nofollow

I left robots.txt unchanged, since with my url structure I'd have to add thousands of * rules, and chatgpt advised against it.

Next, I plan to add few backlinks and check how indexation will change.
 
sensei228 said:
Thanks for the advice!

Here's what I've implemented:
  • Set X-Robots-Tag = 'noindex, nofollow' for comparisons that aren't in top1%.
  • Suggested comparisons now link only to other top1% comparisons.
  • Generated and submitted a sitemap containing only top1% pages.
  • The rest of comparisons are still accessible via manual filtering, and suggestion links on such pages are nofollow

I left robots.txt unchanged, since with my url structure I'd have to add thousands of * rules, and chatgpt advised against it.

Next, I plan to add few backlinks and check how indexation will change.
Click to expand...
Yeah that’s solid. One thing tho—watch out for internal link juice dying off when you noindex + nofollow stuff. If some of those pages are hubs, it might mess with your internal PR flow more than you’d expect.
 
Focus your crawl budget by indexing only high-potential pages (top 1% pairs) and applying noindex to low-value ones to improve SEO efficiency.
 
Last edited:
You're not alone — indexing Programmatic SEO websites correctly is one of the most common challenges,
 
Your concerns are valid. With millions of pages, it’s best to noindex low-value comparisons and focus on indexing only your top, most relevant pages. Keep your sitemap limited to these valuable pages, improve internal linking, and monitor Google Search Console. This helps save crawl budget and improves SEO performance.
 
Beginner in SEO launched a programmatic comparison site, but Google’s indexing odd pages; thinking of noindexing low-value pairs any advice or similar experiences
 
Use noindex on the low-value pairs. This tells Google: “Don’t show these in search results. ”You can still allow crawling if you want Google to follow links on those pages, but noindex is a good start.. Alternatively, disallow crawling via robots.txt, but that prevents Google from seeing the page content/link structure, which can be problematic if those pages link to important pages.
 
congrats on going live! your concerns make sense. focusing indexing on top product pairs and using noindex for low-value pages sounds like a smart move to save crawl budget.
 
sensei228 said:
Hello everyone.
Just for context, I'm a beginner in webdev and SEO, and this is my first project live.

It is programmatic SEO website based on comparisons (Product A vs Product B).

A few days ago I reached my first small milestone: my site started getting indexed.
But what's concerning is that Google started indexing not the pages I was expecting... (obviously my fault.)


How I'm handling indexing right now:
  • Added most content-rich comparison pages to sitemap (fewer than 100).
  • Each comparison page includes suggested comparisons, based on Product A and Product B. Suggestions are each time generated randomly. They are do-follow.
Currently total number of possible pairs is around 2 million pages.
And I'm afraid to waste all crawling budget for pages that won't drive any traffic.

To be honest, I think 99% of these comparisons wouldn't even make sense for people to search.
Most interest is likely in most popular products.
I believe the most of traffic potential lies in comparisons between top 1% pages from A and the top 1% pages from B.


A possible solution I'm considering is adding noindex to all of non-top pairs.


It's hard to say exactly what my question is - I'm just pretty confused rn.
I'd appreciate any advice, and would love to know if my concerns make sense, or where am I wrong.
Click to expand...
Add noindex to low-value pages outside the top product pairs. Focus indexing on the highest-quality, highest-potential comparisons. Random internal links can confuse Google — make them more strategic or remove them. Also, continue refining your sitemap to only include valuable pages.


This will protect your crawl budget and improve SEO over time.
 
if you pSEO is very general and google's AI answering such questions you better don't expect to get it indexed .. after AI's serp answers most of the shit pSEO sites are rekt
 
Beginner here launched a comparison site with 2M possible pages, but Google’s indexing low value ones; should I noindex non-top pairs to save crawl budget?
 
Beginner launching a programmatic SEO site with 2M product comparison pages—Google’s indexing unexpected pages, and I’m worried about crawl budget waste and low-value content; should I noindex non-top pairs?
 
sensei228 said:
Hello everyone.
Just for context, I'm a beginner in webdev and SEO, and this is my first project live.

It is programmatic SEO website based on comparisons (Product A vs Product B).

A few days ago I reached my first small milestone: my site started getting indexed.
But what's concerning is that Google started indexing not the pages I was expecting... (obviously my fault.)


How I'm handling indexing right now:
  • Added most content-rich comparison pages to sitemap (fewer than 100).
  • Each comparison page includes suggested comparisons, based on Product A and Product B. Suggestions are each time generated randomly. They are do-follow.
Currently total number of possible pairs is around 2 million pages.
And I'm afraid to waste all crawling budget for pages that won't drive any traffic.

To be honest, I think 99% of these comparisons wouldn't even make sense for people to search.
Most interest is likely in most popular products.
I believe the most of traffic potential lies in comparisons between top 1% pages from A and the top 1% pages from B.


A possible solution I'm considering is adding noindex to all of non-top pairs.


It's hard to say exactly what my question is - I'm just pretty confused rn.
I'd appreciate any advice, and would love to know if my concerns make sense, or where am I wrong.
Click to expand...
Happens a lot with programmatic sites. Focus your sitemap only on the top useful pairs and set the rest to noindex. Keep internal links clean and avoid random links — that helps Google crawl the right pages first.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top