Google crawling: This Google patent about crawling websites is really interesting

[Castamir]

Google crawling is the first process of any SEO campaign. I recently read something about it that I found really intriguing. According to it, an embodiment of the crawling process carried out by Google are prioritized by domain authority, but this begs the question, "where does that leave time sensitive and important documents on low authority websites?". I believe crawling should be a basic right for every web page. What do you think?

Here's the text to the claim 1 of the patent "System and method for efficient web crawling using domain ranking" (US Patent No. 10,052,542):

1. A computer-implemented method for efficient web crawling, the method comprising: receiving a seed set of Uniform Resource Locators (URLs); determining, for each URL in the seed set, a domain score based on a domain ranking algorithm, wherein the domain ranking algorithm scores a domain based on a plurality of factors comprising: domain age, number of links to the domain, number of unique links to the domain, and domain reputation; prioritizing the URLs based on the domain scores to generate a prioritized list of URLs to crawl; crawling the URLs in the prioritized list according to the order of the prioritized list; and repeating the prioritizing and crawling until a stopping criterion is met, wherein the stopping criterion comprises at least one of: a predetermined number of URLs have been crawled, a predetermined amount of time has elapsed, or no more URLs in the prioritized list meet a relevance threshold.

"

LAYMAN'S EXPLANATION:

• The patent pertains to a computer-implemented method that optimizes Google crawling of the web and indexing its content.
• The method begins by receiving a seed set of URLs that serve as starting points for Google crawling web pages.
• For each URL in the seed set, the method calculates a domain score using a domain ranking algorithm that considers factors like domain age, number of links to the domain, number of unique links to the domain, and domain reputation.

 

[Bugfisch]

Well, their way makes sense - Prioritizing doesn't mean, it's crawling only the big one, but for example it's using 5% of it's crawling requests for the top 1%, but only 5% for the bottom 20%. They still get crawled, espacially initial - but the bot won't show up that often to recheck.
And since googlebots alone produce about 30gb of my nginx log files daily, I don't see that google is not crawling sites with low authority.

 

[resi168]

hello sir

 

[Castamir]

Prioritizing doesn't mean, it's crawling only the big one

Yes it makes sense, but that means a lot of pages from low authority websites never go live on Google search on time. At this age of the internet, they should have come up with a superior tech for that

 

[David2023]

Sure, it makes logic, but it also implies that many pages from websites with low authority never appear in Google search on schedule. In the era of the internet, they ought to have developed a better technology for that.

 

[phone unlockers]

Yes. It's very interesting. I didn't know that. Thanks

 

[Bugfisch]

Yes it makes sense, but that means a lot of pages from low authority websites never go live on Google search on time.

Yeah - but this mostly hits updates, the initial crawl normally comes fast. But yeah, you're right, and Bing does kind of what you're saing. They index first and than punish after 2-4 days on automated regulation much harder than google.

 

[Castamir]

Sure, it makes logic, but it also implies that many pages from websites with low authority never appear in Google search on schedule. In the era of the internet, they ought to have developed a better technology for that.

Another angle, but this one is tricky because it is a race to the top and whoever outdid SEO should rank highest. However, if a system and method is invented by Google for a level playing field, it would be great. I mean, Google can't claim to have inclusive algo that favors small businesses if they're constantly favoring ancient popular websites on search results.

Tiktok has a better inclusive and small business friendly algo than most of the big tech companies, probably except Amazon.

 

[HustleTong]

Whatever tactics they implement, they'll get it twisted at the end