Need to scrape google search URLs, any advice?

hreyhrey · Dec 7, 2021

I've been trying to find a quick and easy way to scrape google search URLs but so far nothing too concrete. I wouldn't need anything too crazy, maybe a few thousand URLs from one search query?

Alexion · Dec 7, 2021

easiest way is through apis like serpapi, or code yourself something fast using selenium so you avoid captchas, or at least minimize it

hreyhrey · Dec 7, 2021

Alexion said:
easiest way is through apis like serpapi, or code yourself something fast using selenium so you avoid captchas, or at least minimize it

Serpapi looks excellent, thanks!

710fla · Dec 7, 2021

You can use Scrapebox it’s a one time payment with lots of uses

Th3 Canadian 3h · Dec 7, 2021

710fla said:
You can use Scrapebox it’s a one time payment with lots of uses

I agree with this and I use scrapebox for everything, kinda of a "all in one tool" for marketers

hreyhrey · Dec 7, 2021

710fla said:
You can use Scrapebox it’s a one time payment with lots of uses

I have scrapebox, but didn't want to go through the proxy headache.

Th3 Canadian 3h · Dec 7, 2021

hreyhrey said:
I have scrapebox, but didn't want to go through the proxy headache.

Then its not that important because 10 of them are only $5

hreyhrey · Dec 8, 2021

Th3 Canadian 3h said:
Then its not that important because 10 of them are only $5

I guess I meant getting the proxy system setup. I’ve only ever done very small scraping with scrape box and didn’t need proxies.

Sweetfunny · Dec 8, 2021

hreyhrey said:
I've been trying to find a quick and easy way to scrape google search URLs but so far nothing too concrete. I wouldn't need anything too crazy, maybe a few thousand URLs from one search query?

It's not possible to obtain a few thousand URLs for one search query unfortunately. Search engine only return "up to" 1,000 results per query even if it says at the top of the search that there millions or billions of results.

And with Google they generally only show 200-600 results per query now days, their theory is if you cant find what you are looking for after a few hundred results you should readjust your query to something more relevant. So you dont need proxies just for one query in ScrapeBox.

Augusto Cury · Dec 9, 2021

Nice, I never heard about scrapbox before, Ill buy it, nice,
some months ago I did my scrapes from google manually with regex, such a troublesome task

karupoiss · Dec 9, 2021

Sweetfunny said:
It's not possible to obtain a few thousand URLs for one search query unfortunately. Search engine only return "up to" 1,000 results per query even if it says at the top of the search that there millions or billions of results.

And with Google they generally only show 200-600 results per query now days, their theory is if you cant find what you are looking for after a few hundred results you should readjust your query to something more relevant. So you dont need proxies just for one query in ScrapeBox.

Where do they mention the max up to 1000 results max only?

plopza · Dec 9, 2021

Augusto Cury said:
Nice, I never heard about scrapbox before, Ill buy it, nice,
some months ago I did my scrapes from google manually with regex, such a troublesome task

Oh yes you will likely find many uses for the tool. It's very lean but does powerful things!

Need to scrape google search URLs, any advice?

hreyhrey

Power Member

Alexion

Banned - Shitlisted and did not do the right thing

hreyhrey

Power Member

710fla

Elite Member

Th3 Canadian 3h

Supreme Member

hreyhrey

Power Member

Th3 Canadian 3h

Supreme Member

hreyhrey

Power Member

Sweetfunny

Elite Member

Augusto Cury

Newbie

karupoiss

Elite Member

plopza

Junior Member

Main Menu

Marketplace

Making Money

BlackHat World