Is there a way to scrape every Google index?

[notmyrealnameoryear88]

Currently I want to publicize a database from a company. all their data is public, but to guess every valid url would be 1x10^83 possible options. So i had the bright idea of doing "site:the url" for that website. This info is denied from the companies API so i thought well if i get every google index thats effectively the same isnt it. Problem is google is a bitch and stops displaying anything after 400. I need around 60 million. Does anyone know a trick around this or is it just an inevitability and i need to think of a new way to get this data?

 

[DeiFic xD]

See if they have a sitemap

 

[notmyrealnameoryear88]

couldnt find that info in the sitemap sadly.

 

[phpbuilt]

Build a site scraper and crawl their website just like Google would.

 

[notmyrealnameoryear88]

Build a site scraper and crawl their website just like Google would.

I did try the site scraper method, but im running into some issue and im just trying to see if there is an alternative before i go this route. The website dynamically loads the results i want, its similar to pagination. you scroll down, more results "products" lets use, the search function will show results seemingly random "type in a" get things starting with a or having an a in it, aswell as i need other languages, some "products" are korean letters. To do this with a site scraper seems nigh impossible from my perspective when i need to get 66 million urls. Or is there a simple thing im missing, those urls aswell are loaded through JS not on the initial html.

 

[notmyrealnameoryear88]

I did try the site scraper method, but im running into some issue and im just trying to see if there is an alternative before i go this route. The website dynamically loads the results i want, its similar to pagination. you scroll down, more results "products" lets use, the search function will show results seemingly random "type in a" get things starting with a or having an a in it, aswell as i need other languages, some "products" are korean letters. To do this with a site scraper seems nigh impossible from my perspective when i need to get 66 million urls. Or is there a simple thing im missing, those urls aswell are loaded through JS not on the initial html.

Update: the website acts the same as google. the search functions only gives 1000 reults before saying no more results.

 

[UnitedSEO]

Currently I want to publicize a database from a company. all their data is public, but to guess every valid url would be 1x10^83 possible options. So i had the bright idea of doing "site:the url" for that website. This info is denied from the companies API so i thought well if i get every google index thats effectively the same isnt it. Problem is google is a bitch and stops displaying anything after 400. I need around 60 million. Does anyone know a trick around this or is it just an inevitability and i need to think of a new way to get this data?

Scraping every Google index is not feasible due to limitations set by Google. The search results are restricted, and you can only view up to 400 pages per query. Even if you use the "site:" operator, you'll still encounter this limitation. Attempting to scrape millions of pages from Google's index is against their terms of service, and it can lead to IP blocking or legal consequences.

If the company's data is public but difficult to access through traditional means, it's essential to explore alternative solutions rather than resorting to unauthorized scraping. Consider reaching out to the company to discuss the possibility of accessing their data through an API or other official channels. Respect for the terms of service and compliance with legal and ethical standards are essential when handling web data.

 

[notmyrealnameoryear88]

Scraping every Google index is not feasible due to limitations set by Google. The search results are restricted, and you can only view up to 400 pages per query. Even if you use the "site:" operator, you'll still encounter this limitation. Attempting to scrape millions of pages from Google's index is against their terms of service, and it can lead to IP blocking or legal consequences.

If the company's data is public but difficult to access through traditional means, it's essential to explore alternative solutions rather than resorting to unauthorized scraping. Consider reaching out to the company to discuss the possibility of accessing their data through an API or other official channels. Respect for the terms of service and compliance with legal and ethical standards are essential when handling web data.

First. no google couldnt take legal action. its been held up in the supreme court that scraping is legal. could i be banned? absolutely, could i be arrested? not in a million years. Second, I'm in a blackhat forum, is this not a blackhat community lmao im confused here, the whole thing with blackhats is not using the expected methods.

 

[GenesisOne]

Scraping every Google index is not feasible due to limitations set by Google. The search results are restricted, and you can only view up to 400 pages per query. Even if you use the "site:" operator, you'll still encounter this limitation. Attempting to scrape millions of pages from Google's index is against their terms of service, and it can lead to IP blocking or legal consequences.

If the company's data is public but difficult to access through traditional means, it's essential to explore alternative solutions rather than resorting to unauthorized scraping. Consider reaching out to the company to discuss the possibility of accessing their data through an API or other official channels. Respect for the terms of service and compliance with legal and ethical standards are essential when handling web data.

Somehow Ahrefs and Semrush do this - so there must be a way.

 

[Ashk881]

Somehow Ahrefs and Semrush do this - so there must be a way.

They don't do that. They crawl your site and just check if each page appears in a google search that looks like "<site url> <page title>" to check if its in the index.

OP, google won't give you more than 500 results for site:

 

[fc-dh]

Scraping every Google index is not feasible due to limitations set by Google. The search results are restricted, and you can only view up to 400 pages per query. Even if you use the "site:" operator, you'll still encounter this limitation. Attempting to scrape millions of pages from Google's index is against their terms of service, and it can lead to IP blocking or legal consequences.

Do you even know where you are?

 

[Steptoe]

Currently I want to publicize a database from a company. all their data is public, but to guess every valid url would be 1x10^83 possible options. So i had the bright idea of doing "site:the url" for that website. This info is denied from the companies API so i thought well if i get every google index thats effectively the same isnt it. Problem is google is a bitch and stops displaying anything after 400. I need around 60 million. Does anyone know a trick around this or is it just an inevitability and i need to think of a new way to get this data?

Unfortunately the site: search modifier is hamstrung, and seems to be getting worse. When I'm trying to get around the 300-400 results limitation I start adding keywords to the query. Not anything relevant, but a combination of stop words (this, that, their etc), single letters/numbers (a, b, 1, 2 etc), negative single letters/numbers (-a, -b, -1, -2 etc), negative stop words (-this, -that, -their etc). These will sometimes give you a significant number of different results, sometimes only a handful.

If the pages the data you're interested is on have a specific url structure you could add some inurl: modifiers to your query to restrict to pages you're interested in. You could even play around with date modifiers like before:2020 and after:2021 - you just might squeeze out some new urls.