Scrape google...

W

William Rufino

Newbie

Aug 3, 2011

32

2

• Sep 28, 2011

• #1

Hello There,

Now google isnt taking the parameter num anymore to get more than 10 results...

Has anyone seen a workaround for that?

 

scraper1

Regular Member

May 28, 2011

214

209

• Sep 28, 2011

• #2

Yes, just turn off Google Instant from Preferences.

 

W

webmast

Regular Member

Dec 18, 2010

337

239

• Sep 28, 2011

• #3

Code:

http://www.google.com/search?q=buy+viagra&num=100

Still works for me

scraper1 said:

Yes, just turn off Google Instant from Preferences.

Click to expand...

 

Last edited: Sep 28, 2011

scraper1

Regular Member

May 28, 2011

214

209

• Sep 28, 2011

• #4

There are more ways to do the same thing:

Code:

http://www.google.com/preferences

 

W

William Rufino

Newbie

Aug 3, 2011

32

2

• Sep 28, 2011
• Thread Starter Thread Starter

• #5

I know that guys, but i'm talking about when programming, o use CURL to crawl google, and preferences save a cookie, how do i retrieve that cookie so i can also use it on curl?

 

scraper1

Regular Member

May 28, 2011

214

209

• Sep 28, 2011

• #6

Check out the Google search API, you can use it with curl

Code:

http://code.google.com/apis/websearch/docs/

Code:

curl -e http://www.my-ajax-site.com \
'https://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=Paris%20Hilton&key=INSERT-YOUR-KEY'

Never mind, it seems that there is a limitation for 10 results only, when using the API.
You can store the cookie in a file, and use it from there.

 

W

William Rufino

Newbie

Aug 3, 2011

32

2

• Sep 29, 2011
• Thread Starter Thread Starter

• #7

the problem with the API is that it shows different results than on the web.... :/

 

S

scriptomania

Junior Member

Dec 28, 2010

126

251

• Oct 1, 2011

• #8

1. Look around through some useragents list
2. Change yours to an old ass one (win98 era)
3. ???
4. Profit!

 

MrBlue

Senior Member

Dec 18, 2009

975

695

• Oct 1, 2011

• #9

William Rufino said:

I know that guys, but i'm talking about when programming, o use CURL to crawl google, and preferences save a cookie, how do i retrieve that cookie so i can also use it on curl?

Click to expand...

I stopped using PHP/cURL to scrape Google. Now using Python...way more powerful and my scripts are now much less brittle.

 

C

christoss1959

Senior Member

Nov 25, 2010

896

1,176

• Oct 1, 2011

• #10

Read this:

Code:

http://www.our-picks.com/archives/2007/01/30/google-search-urls-revealed-or-how-to-create-your-own-search-url/

 

W

William Rufino

Newbie

Aug 3, 2011

32

2

• Oct 2, 2011
• Thread Starter Thread Starter

• #11

I use python too, but i uyse PyCURL + beautifulsoup

i'll check a list of old user agents to try

 

MrBlue

Senior Member

Dec 18, 2009

975

695

• Oct 2, 2011

• #12

William Rufino said:

I use python too, but i uyse PyCURL + beautifulsoup

i'll check a list of old user agents to try

Click to expand...

Creator of Beautifulsoup isn't supporting it anymore. Take a look at lxml!

 

W

William Rufino

Newbie

Aug 3, 2011

32

2

• Oct 2, 2011
• Thread Starter Thread Starter

• #13

seriously? didn't know that! but as for now beautifulsoup is pretty awesome...

And it was just a matter of user agent! hehe fixed it now thx guys

 

xenon2010

Regular Member

Apr 27, 2010

233

49

• Oct 17, 2011

• #14

yea just use random user-agents and you will be fine...

 

O

omega369

BANNED

Sep 20, 2015

37

17

• Jul 16, 2018

• #15

/Up

 

M

maxsch0610

Newbie

Jul 27, 2018

19

4

• Jul 30, 2018

• #16

try use proxies as on google you get fast a ban and need to verify that you are a human by recapthca

 

T

theRevolt

Jr Vip

Jul 29, 2009

2,925

2,139

• Jul 30, 2018

• #17

maxsch0610 said:

try use proxies as on google you get fast a ban and need to verify that you are a human by recapthca

Click to expand...

You see this thread is 7 years old?

 

M

maxsch0610

Newbie

Jul 27, 2018

19

4

• Jul 30, 2018

• #18

theRevolt said:

You see this thread is 7 years old?

Click to expand...

I didn't saw , sry