Site Indexing

[seomasterforyou]

Hello all the members of the BlackHatWorld can anyone tell what is the best way to get my site indexed my website has nearly 500000 pages and only 100000 of them are indexed how can i make the rest of pages get indexed on google

 

[Ashk881]

Hello all the members of the BlackHatWorld can anyone tell what is the best way to get my site indexed my website has nearly 500000 pages and only 100000 of them are indexed how can i make the rest of pages get indexed on google

It says right there. Click on that info button to know the 8 reasons.

The right way to get so many pages indexed is to improve your site reputation. You can do that by:
- bringing more quality backlinks
- through reputed author tags in posts
- mentions/contextual links etc

Google sets a crawl budget depending on the the reputation and activity of a website. Take this website for example. Almost every page on BHW is indexed. That's because Google believes it's a high quality site with very frequent activity and sets a very high crawl budget for this site.

 

[seomasterforyou]

It says right there. Click on that info button to know the 8 reasons.

The right way to get so many pages indexed is to improve your site reputation. You can do that by:
- bringing more quality backlinks
- through reputed author tags in posts
- mentions/contextual links etc

Google sets a crawl budget depending on the the reputation and activity of a website. Take this website for example. Almost every page on BHW is indexed. That's because Google believes it's a high quality site with very frequent activity and sets a very high crawl budget for this site.

Well appreciate your reply, but my website DA is 63, and what exactly is happening with my website is that one month, a large number of pages get indexed, and in the next, all of them suddenly start deindexing, so I am searching for a short-term technique or any tool that can help my website get indexed again

 

[Ashk881]

Well appreciate your reply, but my website DA is 63, and what exactly is happening with my website is that one month, a large number of pages get indexed, and in the next, all of them suddenly start deindexing, so I am searching for a short-term technique or any tool that can help my website get indexed again

1. Google does not use DA for calculating *site reputation*. Your DA would be 80 but for Google it might as well be equivalent to 0

2. The pages get deindexed because Google thinks either the pages are spam or thin-content. So you probably want to fix that

3. There are tools out there that do this. Just google for "index site saas" etc and you will find a dozen. But I wouldn't recommend using them. Most of them only ping googles index request api or put links to your pages on their websites which Google frequently crawls.

 

[SunnFun]

Hello all the members of the BlackHatWorld can anyone tell what is the best way to get my site indexed my website has nearly 500000 pages and only 100000 of them are indexed how can i make the rest of pages get indexed on google

To index the remaining pages of your website, segment your sitemap into multiple files, each containing up to 50,000 URLs, and submit them via Google Search Console. Ensure all URLs are canonical and not blocked by robots.txt or marked with "noindex" tags. Utilize the URL Inspection Tool in Google Search Console to request indexing for individual pages or groups of pages. By implementing these steps, you can enhance the visibility and indexing of your site's pages

 

[seomasterforyou]

To index the remaining pages of your website, segment your sitemap into multiple files, each containing up to 50,000 URLs, and submit them via Google Search Console. Ensure all URLs are canonical and not blocked by robots.txt or marked with "noindex" tags. Utilize the URL Inspection Tool in Google Search Console to request indexing for individual pages or groups of pages. By implementing these steps, you can enhance the visibility and indexing of your site's pages

Already did that. Nothing Helps

 

[Ranking Expert]

You can also try Indexing Services from Marketplace.

 

[seomasterforyou]

You can also try Indexing Services from Marketplace.

Can you suggest some that are worth trying because i have tried Indexification and it didn't help

 

[tazarbm]

honest question: are the non-indexed pages worth indexing? Because if they're not, I agree with @Ashk881 in that google tends to refuse to index low quality pages.

Or, if the pages are something like expired job listings then there's no point in indexing them (in what google's concerned).

If they are indeed valid pages you can try a paid indexer, or point some internal links to them from other pages on your site (that are highly traffic'd and / or rank well in google preferably).... you can use a tool like LinkWhisper to automate the internal linking as I'm pretty sure you don't want to do this manually for 100k pages

And again like Ashk said, check the reasons that google have given you, many times those reasons are enough to fix the issue if it's stuff related to sitemaps, crawl errors, redirects, canonicalization, noindex tags, etc... things that can easily be fixed on your own in a few minutes...

Also, if these pages use 100% AI content then it's possible that google refuses to index them due to low quality / thin content, they've been trying a lot to get rid of low quality AI content lately, so you might want to update / improve that content if it's 100% AI and unedited...

Last but not least, it could be a google issue, in which case there's nothing for you to do but wait for them to fix it. Lately, they've been having indexing issues periodically, so it's possible that it's this... although, if it was google's fault then I don't think they'd warn you about 8 indexation issues they've found with those pages, so - once again - the problem is most likely on your site, so start with that!

 

[seomasterforyou]

honest question: are the non-indexed pages worth indexing? Because if they're not, I agree with @Ashk881 in that google tends to refuse to index low quality pages.

Or, if the pages are something like expired job listings then there's no point in indexing them (in what google's concerned).

If they are indeed valid pages you can try a paid indexer, or point some internal links to them from other pages on your site (that are highly traffic'd and / or rank well in google preferably).... you can use a tool like LinkWhisper to automate the internal linking as I'm pretty sure you don't want to do this manually for 100k pages

And again like Ashk said, check the reasons that google have given you, many times those reasons are enough to fix the issue if it's stuff related to sitemaps, crawl errors, redirects, canonicalization, noindex tags, etc... things that can easily be fixed on your own in a few minutes...

Also, if these pages use 100% AI content then it's possible that google refuses to index them due to low quality / thin content, they've been trying a lot to get rid of low quality AI content lately, so you might want to update / improve that content if it's 100% AI and unedited...

Last but not least, it could be a google issue, in which case there's nothing for you to do but wait for them to fix it. Lately, they've been having indexing issues periodically, so it's possible that it's this... although, if it was google's fault then I don't think they'd warn you about 8 indexation issues they've found with those pages, so - once again - the problem is most likely on your site, so start with that!

Thanks for your reply, but the most number of pages are in the Crawled but Currently not Indexed and Discovered but currently not indexed so I do not think there is any error in the website

 

[tazarbm]

Thanks for your reply, but the most number of pages are in the Crawled but Currently not Indexed and Discovered but currently not indexed so I do not think there is any error in the website

so it looks like a content quality or internal linking issue then. I can't think of anything else...

 

[proxygo]

Hello all the members of the BlackHatWorld can anyone tell what is the best way to get my site indexed my website has nearly 500000 pages and only 100000 of them are indexed how can i make the rest of pages get indexed on google

To get your site indexed by search engines like Google,
focus on creating high-quality content, optimizing your
site structure, and submitting a sitemap to Google Sear
ch Console. Regularly update your content, ensure
your site is crawlable, and monitor your site's
indexing status using Search Console.

 

[OwenReed]

Look into log file analysis to see how Googlebot is crawling your site. It'll show what areas are being ignored or crawled inefficiently.

 

[seomasterforyou]

Look into log file analysis to see how Googlebot is crawling your site. It'll show what areas are being ignored or crawled inefficiently.

Thank you for the reply
Can you tell where i can see this log file analysis

 

[seomasterforyou]

Thank you for the reply
Can you tell where i can see this log file analysis or are you asking about Crawl stats report in GSC

 

[bartosimpsonio]

Mate you need to do some basic research

Add your site to search console and see which errors are causing deindexing

 

[seomasterforyou]

Mate you need to do some basic research

Add your site to search console and see which errors are causing deindexing

Thanks for your reply, but the most number of pages are in the Crawled but Currently not Indexed and Discovered but currently not indexed so I do not think there is any error in the website

Thanks for you reply Sir
But if you will read the thread carefully you will see the answer

 

[Steptoe]

Hello all the members of the BlackHatWorld can anyone tell what is the best way to get my site indexed my website has nearly 500000 pages and only 100000 of them are indexed how can i make the rest of pages get indexed on google

What is the content of the 400k pages not listed as indexed? Is it thin, recycled product descriptions, or rich unique content? In the current climate, having 100k pages indexed is pretty good going.

 

[seomasterforyou]

What is the content of the 400k pages not listed as indexed? Is it thin, recycled product descriptions, or rich unique content? In the current climate, having 100k pages indexed is pretty good going.

Well this is the site you can check it out this is a astrology site https://www.alokikastro.com/ and i think it has rich unique content in every page and i have tried so many things but the rest pages are getting not indexed and its DA is 9

 

[Steptoe]

After some limited checks there seems to be a lot of repeated or barely edited content throughout all the pages, not what I would call rich unique content.