Scrape google...

[William Rufino]

Hello There,

Now google isnt taking the parameter num anymore to get more than 10 results...

Has anyone seen a workaround for that?

 

[scraper1]

Yes, just turn off Google Instant from Preferences.

 

[webmast]

http://www.google.com/search?q=buy+viagra&num=100

Still works for me

Yes, just turn off Google Instant from Preferences.

 

[scraper1]

There are more ways to do the same thing:

http://www.google.com/preferences

 

[William Rufino]

I know that guys, but i'm talking about when programming, o use CURL to crawl google, and preferences save a cookie, how do i retrieve that cookie so i can also use it on curl?

 

[scraper1]

Check out the Google search API, you can use it with curl

http://code.google.com/apis/websearch/docs/

curl -e http://www.my-ajax-site.com \
'https://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=Paris%20Hilton&key=INSERT-YOUR-KEY'

Never mind, it seems that there is a limitation for 10 results only, when using the API.
You can store the cookie in a file, and use it from there.

 

[William Rufino]

the problem with the API is that it shows different results than on the web.... :/

 

[scriptomania]

1. Look around through some useragents list
2. Change yours to an old ass one (win98 era)
3. ???
4. Profit!

 

[MrBlue]

I know that guys, but i'm talking about when programming, o use CURL to crawl google, and preferences save a cookie, how do i retrieve that cookie so i can also use it on curl?

I stopped using PHP/cURL to scrape Google. Now using Python...way more powerful and my scripts are now much less brittle.

 

[christoss1959]

Read this:

http://www.our-picks.com/archives/2007/01/30/google-search-urls-revealed-or-how-to-create-your-own-search-url/

 

[William Rufino]

I use python too, but i uyse PyCURL + beautifulsoup

i'll check a list of old user agents to try

 

[MrBlue]

I use python too, but i uyse PyCURL + beautifulsoup

i'll check a list of old user agents to try

Creator of Beautifulsoup isn't supporting it anymore. Take a look at lxml!

 

[William Rufino]

seriously? didn't know that! but as for now beautifulsoup is pretty awesome...

And it was just a matter of user agent! hehe fixed it now thx guys

 

[xenon2010]

yea just use random user-agents and you will be fine...

 

[omega369]

/Up

 

[maxsch0610]

try use proxies as on google you get fast a ban and need to verify that you are a human by recapthca

 

[theRevolt]

try use proxies as on google you get fast a ban and need to verify that you are a human by recapthca

You see this thread is 7 years old?

 

[maxsch0610]

You see this thread is 7 years old?

I didn't saw , sry