SerpAPI alternatives

I've been using SerpApi to scrape emails from Google based on hashtags and all social medias.

For each query I can only get around 300 results before it chokes out. So I need to rotate a ton of variables to make new results show up on the first 30 pages (3 pages of 100 results to reduce API calls).

Is there any SerpAPI competitors that can go further per query? I know it's a very specific issue but any insight would be greatly appreciated. Also if SerpAPI has a way to go further that'd also work.

Ty!

The 300 results wall will come from Google itself. Most queries I find even though they say there are thousands of results available, Google will stop showing results after 300-500.
You can potentially go further by adding stop words to your queries (eg. a, and, this, that), which may have the effect of giving you a different set of results.

SerpApi is pricey though. Something like ScaleSerp or ScraperAPI should be able to give you the same data for less $$

gabasiq said:

I've been using SerpApi to scrape emails from Google based on hashtags and all social medias.

For each query I can only get around 300 results before it chokes out. So I need to rotate a ton of variables to make new results show up on the first 30 pages (3 pages of 100 results to reduce API calls).

Is there any SerpAPI competitors that can go further per query? I know it's a very specific issue but any insight would be greatly appreciated. Also if SerpAPI has a way to go further that'd also work.

Ty!

Click to expand...

As @Steptoe suggested, the limitation is from Google, it won't go past page 29-33.

I know because I coded a Google Scraper for Instagram profiles, for the same reason you said(emails) for a client, and he was frustrated because of that limit. He thought it was because of me, took me days to explain to him ))

You really have to mix variables up, like changing first names.
e.g. site:instagram.com "@gmail.com" "New york" john

Alexion said:

As @Steptoe suggested, the limitation is from Google, it won't go past page 29-33.

I know because I coded a Google Scraper for Instagram profiles, for the same reason you said(emails) for a client, and he was frustrated because of that limit. He thought it was because of me, took me days to explain to him ))

You really have to mix variables up, like changing first names.
e.g. site:instagram.com "@gmail.com" "New york" john

Click to expand...

Well, it's nice to know it wasn't the API itself or my code getting broken. Lol yeah, older clients complain to me when they forget their Facebook password like it's my fault, I know the drama.
Yeah I've been switching the time filters+hashtags+social medias+emails+languages, but I'll include the stop words like Steptoe recommended and that should 5X the amount per keyword. Good stuff.

Steptoe said:

The 300 results wall will come from Google itself. Most queries I find even though they say there are thousands of results available, Google will stop showing results after 300-500.
You can potentially go further by adding stop words to your queries (eg. a, and, this, that), which may have the effect of giving you a different set of results.

SerpApi is pricey though. Something like ScaleSerp or ScraperAPI should be able to give you the same data for less $$

Click to expand...

That's very helpful. I was using time filters, 1 day ago, 1 week ago, 1 month ago etc. But the stop words is really smart, I'll definitely use that.
I'll check those API's as well, I appreciate the solid advice!

Big list of stop words for you attached:

Or use google api itself??

"

Pricing​

Custom Search JSON API provides 100 search queries per day for free. If you need more, you may sign up for billing in the API Console. Additional requests cost $5 per 1000 queries, up to 10k queries per day.

"

https://developers.google.com/custom-search/v1/overview