How to scrape top 10 results in Google in bulk?

ardley216 · Aug 24, 2014

Hi,

Basically as the title says, is there any way to scrape the top 10 urls for search terms on mass?

Or if anyone knows a service that supplies this?

Thanks!

ticalico · Aug 24, 2014

Hmm...

You could maybe right a program in php and use CURL.

It would scrape the whole page and you could have it return JUST content of a specific ID or Class.

Target those top 10 ignoring or not ignoring the advertisements also listed in the page.

Put it in a list etc....

Schvamp · Aug 24, 2014

I'm sure it can be done with scrapebox, with the right settings, unfortunately I ain't that familiar with it tho.
If you need a custom scraper, shoot me a PM.

himanuzo · Aug 24, 2014

Market Samurai software can do it.

ardley216 · Aug 24, 2014

Hi,

Thanks for your suggestions.

I will be needing this to scrape tens of thousands of search results everyday.

macdonjo3 · Aug 24, 2014

I've used this: https://github.com/MarioVilas/google

himanuzo · Aug 24, 2014

ardley216 said:
Hi,

Thanks for your suggestions.

I will be needing this to scrape tens of thousands of search results everyday.

For this, you need to use many paid proxies monthly. Because Google doesn't allow it.

macdonjo3 · Aug 24, 2014

himanuzo said:
For this, you need to use many paid proxies monthly. Because Google doesn't allow it.

Actually, with proper intervals, he won't need many at all. I tested this last year... unless things have changed since then.

Pejlis · Aug 24, 2014

you cant get thousands they dont allow you

Schvamp · Aug 24, 2014

himanuzo said:
For this, you need to use many paid proxies monthly. Because Google doesn't allow it.

Pejlis said:
you cant get thousands they dont allow you

Lets say OP needs to scrape 30,000 urls a day.
30000 / 10 results per page = 3000 requests.
3000 / 24h a day = 125 requests per hour.
125 / 60 minutes per hour = ~2 requests/min.

2 requests per minute, not possible?
Easy to scale with proxy rotation,vps and multithreading.

ardley216 · Aug 24, 2014

macdonjo3 said:
I've used this: https://github.com/MarioVilas/google

Thanks for that. How does it work? Luckily my business partner is learning python right now.

Schvamp said:
Lets say OP needs to scrape 30,000 urls a day.
30000 / 10 results per page = 3000 requests.
3000 / 24h a day = 125 requests per hour.
125 / 60 minutes per hour = ~2 requests/min.

2 requests per minute, not possible?
Easy to scale with proxy rotation,vps and multithreading.

Yeah that sounds good. Except, I'll be doing 300k a day.

Is there a php script that uses CURL, like ticalico mentioned? One that does proxy rotation?

SeoQuest · Aug 25, 2014

In Scrapebox in Results field type 10 and it will scrape the first 10 urls for each keyword in your list.

prab1996 · Aug 25, 2014

this can be easily done with sb.

if you can't pay then try gscraper free and set it to scrape only top 10 results.
-=-

mindmaster · Aug 25, 2014

Yes, Scrapebox can do that easily just as SeoQuest and Prab said.

MissioneR · Aug 25, 2014

SeoQuest said:
In Scrapebox in Results field type 10 and it will scrape the first 10 urls for each keyword in your list.

It's the simple method where you can get what you want.

ardley216 · Oct 29, 2014

MissioneR said:
It's the simple method where you can get what you want.

Yes, except I would need it to be put straight into a database. and I am looking at 6000 requests day.

How to scrape top 10 results in Google in bulk?

ardley216

Elite Member

ticalico

Registered Member

Schvamp

Power Member

himanuzo

Supreme Member

ardley216

Elite Member

macdonjo3

Elite Member

himanuzo

Supreme Member

macdonjo3

Elite Member

Pejlis

Regular Member

Schvamp

Power Member

ardley216

Elite Member

SeoQuest

Junior Member

prab1996

Elite Member

mindmaster

Elite Member

MissioneR

Elite Member

ardley216

Elite Member

Main Menu

Marketplace

Making Money

BlackHat World