How to increase the crawl & index rate for my new site?

malayan · Jul 19, 2022

Here's the skinny:

I'm a formerly retired SEO, back at it.

This website is a complete biz in a competitive and regulated finance space.

So, just to give you an idea, site has: About Us | Staff | Ethics Pages | Privacy/Terms/Disclaimer | Contact Us | a news section, services, articles, etc... It's not a thin content farm site. It even has an account registration process w/ different user pathways, etc...

The site also has no affiliate links, ads, or references to anything affiliate based.

BUT, the site is under 30 days old and I'm trying to grow it quickly.

I've been adding highly technical content to the site, at a rate of about 7 per day, several times a week. Each post is structured with a decent schema and high text-to-html ratio (+30%), and every post is at least 500 words (some as large as 3000 words). The site's internal linking is pretty decent and the load speed is decent. The site also has a social footprint in everything from Crunchbase to Facebook, it even has its own subreddit, but I can't seem to increase the indexing rate.

FYI: I just added and implemented the Rank Math Instant Indexer w/ the Google API, right before posting this.

Do you guys have any ideas what else I can do to increase the index rate?

Quite a few of the pages are "Crawled - currently not index".

Right now the Google search: site: mydomain.com only retrieves 29 out of nearly 85 pages.

[attaching screenshots for reference]

malayan · Jul 19, 2022

So many readers and no ideas?

PHPInjected · Jul 19, 2022

malayan said:
So many readers and on ideas?

Outside of the noindex tags, you've done everything seemingly right. Which pages of yours are tagged with a noindex tag? Sample pages & author pages?

malayan · Jul 19, 2022

PHPInjected said:
Outside of the noindex tags, you've done everything seemingly right. Which pages of yours are tagged with a noindex tag? Sample pages & author pages?

Yeah, I figured out the noindex part after the fact. I realized I need to keep the "nofollows" in the robots.txt

PHPInjected · Jul 19, 2022

malayan said:
Yeah, I figured out the noindex part after the fact. I realized I need to keep the "nofollows" in the robots.txt

The age of the domain is most likely your issue or at least a contributing factor. You're still very much inside of a sandbox stage because your website isn't even 30 days old. Even though that only pertains to your positions in the SERPs.

Indexing is flakey AT BEST right now. Most people on the forum have experienced this at some point. Have you tested the Youtube live stream method on any of your pages?

Alexion · Jul 19, 2022

Try this
https://www.blackhatworld.com/seo/heres-how-to-index-your-stuff-using-reddit.1409943/#post-15295077
might help

malayan · Jul 19, 2022

Alexion said:
Try this
https://www.blackhatworld.com/seo/heres-how-to-index-your-stuff-using-reddit.1409943/#post-15295077
might help

Ahh, the cross-posting...

Good idea.

Thank you buddy.

Alexion · Jul 19, 2022

malayan said:
Ahh, the cross-posting...

Good idea.

Thank you buddy.

No no, don't cross post. Please read the thread, it's not cross posting.
You actually have to copy paste the links again in your sub.

malayan · Jul 19, 2022

Alexion said:
No no, don't cross post. Please read the thread, it's not cross posting.
You actually have to copy paste the links again in your sub.

"The thing is, when you repost something, there's going to be a backlink to your post from the huge one. It shows in the bottom right where it says "View this discussion in X more places"."

This is what I was referring to.

I guess my terminology was incorrect, thank you anyway.

It sounds like this is just to get crawlers to read my subreddit, I wouldn't want to risk being banned, flamed, or downvoted to heck by crossposting every post from my r/

malayan · Jul 20, 2022

PHPInjected said:
The age of the domain is most likely your issue or at least a contributing factor. You're still very much inside of a sandbox stage because your website isn't even 30 days old. Even though that only pertains to your positions in the SERPs.

Indexing is flakey AT BEST right now. Most people on the forum have experienced this at some point. Have you tested the Youtube live stream method on any of your pages?

The domain's too new to even set up an associated YouTube channel w/ the Google Suite account. It'll be 30 days old in five days and I'll be able to associate an account w/ it.

UPDATE: As of today the site went from 28 indexed pages to 43 indexed pages.

I'm honestly considering just ONLY focusing on YouTube SEO & video production at this point, it doesn't make sense to invest this much money in content that Google's not even going to bother indexing. Especially when being in finance means that I'm going to have to contend with YMYL guidelines in the near future.

malayan · Jul 20, 2022

Indexing update: I started to focus on updating existing pages, improving the internal linking structure, and restructuring the categorizing of the posts (wordpress site).

malayan · Jul 20, 2022

Also interesting...

I created this page around 7pm local time and it was crawled and indexed in minutes.

It's about 1,000 words long and has an improved internal linking structure.

brandwizz · Jul 20, 2022

If your target is to increase only the crawl rate & indexing then there are several things that you need to understand first on what basis are your pages being crawled frequently by the crawler.
1. Page load speed- is the one thing that you have to optimize because, if your page loading speed is usualy higher then crawler will not crawled them properly and they return back
2. Internal linking- as they important that gives authority and discover new pages easily
3. Backlink- this one is also important because it give authority to your pages.
4. Search Console- there is a hidden method from which you can change the current crawl rate of your website.(https://support.google.com/webmasters/answer/48620?hl=en)

malayan · Jul 21, 2022

It looks like the site's getting indexed faster now.

malayan · Jul 22, 2022

I guess this thread is dead, but I'll cap it off with this:

Google's now indexing EVERY page.

It looks like it was just because the site is new.

jokerpablo01 · Jul 25, 2022

PHPInjected said:
The age of the domain is most likely your issue or at least a contributing factor. You're still very much inside of a sandbox stage because your website isn't even 30 days old. Even though that only pertains to your positions in the SERPs.

Indexing is flakey AT BEST right now. Most people on the forum have experienced this at some point. Have you tested the Youtube live stream method on any of your pages?

what problems arise in the older age area

malayan · Jul 31, 2022

Update: Google's actually removed over 100 pages from the index. GSC says they're still indexed, but searching for "site:_________.com" reveals less everytime.

I went from over 100 indexed to 21 indexed.

Tam1990 · Jul 31, 2022

malayan said:
Update: Google's actually removed over 100 pages from the index. GSC says they're still indexed, but searching for "site:_________.com" reveals less everytime.

I went from over 100 indexed to 21 indexed.

I have faced same issue now for months.. The page that shows indexed in GSC, look them up by searching site:abc.com/full-url and see if it is indexed or not, I think I might have solution for that..

malayan · Jul 31, 2022

Tam1990 said:
I have faced same issue now for months.. The page that shows indexed in GSC, look them up by searching site:abc.com/full-url and see if it is indexed or not, I think I might have solution for that..

So, what's the solution?

Tam1990 · Jul 31, 2022

malayan said:
So, what's the solution?

Did you even check if it is indexed? Also check Mobile Usability in GSC and let me know how many valid pages it shows..

How to increase the crawl & index rate for my new site?

Power Member

Power Member

Super Moderator

Power Member

Attachments

Super Moderator

Banned - Shitlisted and did not do the right thing

Power Member

Banned - Shitlisted and did not do the right thing

Power Member

Power Member

Power Member

Power Member

Attachments

Newbie

Power Member

Attachments

Power Member

Regular Member

Power Member

Regular Member

Power Member

Regular Member

Main Menu

Marketplace

Making Money

BlackHatWorld

Other