Parse google with python...

W

William Rufino

Newbie

Aug 3, 2011

32

2

• Aug 18, 2011

• #1

Hello everyone,

I'm trying to make a bot, but i'm getting a little problem with pycurl and google is giving me a 400 error code, but when i do it in php it actually works.

I'm even using the same user agent.. I already have it in php but i wanna port it to python to use multi thread

 

M

Mosquera

Newbie

Feb 24, 2009

32

18

• Aug 19, 2011

• #2

Any more info? There are times where google says you're a bot and asks for you to complete a captcha.

I'm not sure what error code it returns in those cases, but it could be 400. I don't remember though, I would check everything before assuming something.

 

C

confined

Regular Member

Jan 4, 2009

216

91

• Aug 19, 2011

• #3

it has to do with scraping frequency, proxies and random sleeps help with that.

 

W

wu1239

Newbie

Jun 4, 2011

16

0

• Aug 23, 2011

• #4

send us more info and we will help you
limited detail won't help you

 

X

Xooor

Newbie

Aug 14, 2011

18

17

• Aug 26, 2011

• #5

William Rufino said:

Hello everyone,

I'm trying to make a bot, but i'm getting a little problem with pycurl and google is giving me a 400 error code, but when i do it in php it actually works.

I'm even using the same user agent.. I already have it in php but i wanna port it to python to use multi thread

Click to expand...

I've got a working Google scraping bot of mine up, I wrote it in PHP using CURL and switches between 25 paid proxies that I have to avoid getting banned by Google.

I also am a great fan of python, I would recommend looking into urllib2 and BeautifulSoup, which are two brilliant python libs I use for writing bots.

If you need any help say so. Wish you good luck