python crawler

777lady · Jan 13, 2015

Hello,

i try to write a python serp crawler with proxy.
I have there now a problem that proxy work in browser but not with the python script, google block the request.

This is the passage in my script

response = request.get(url, proxies=proxies, headers=headers, verify=True)

Can anyone helpme?

Thanks

Lady

bartosimpsonio · Jan 13, 2015

Are you changing the user agent string? Google will block user agent strings from default python libraries.

777lady · Jan 13, 2015

yes i do i have about 20 user agents which i everytime also change.
Is there mabye some user agents which google do not like?

myopic1 · Jan 13, 2015

777lady said:
Hello,

i try to write a python serp crawler with proxy.
I have there now a problem that proxy work in browser but not with the python script, google block the request.

This is the passage in my script

response = request.get(url, proxies=proxies, headers=headers, verify=True)

Can anyone helpme?

Thanks

Lady

You are correct, proxies do not work from terminal unless you specifically incorporate it in your code to do so.

Simply do the following:

arrayOfProxies = ["xxx.xxx.xxx.xxx", "xxx.xxx.xxx.xxx"]
for x in arrayOfProxies:

r = requests.get(url, headers=headers, proxies=x)

777lady · Jan 13, 2015

I try this also, but same effect.

TheVegan · Jan 13, 2015

First I would say just don't use google, they're pretty good at blocking.. I would use yahoo or some other random search engine... Buuuut, what you can also use is Tor browser, if your using python it's pretty easy to connect via tor.

tony_d · Jan 14, 2015

TheVegan said:
First I would say just don't use google, they're pretty good at blocking.. I would use yahoo or some other random search engine... Buuuut, what you can also use is Tor browser, if your using python it's pretty easy to connect via tor.

That would be about as useful as a chocolate teapot... who wants a serp checker that just checks yahoo? Nobody.

seoproranker · Jan 14, 2015

Few public proxies will work against G

you need to use dedicated private proxies

and you are not in a hurry Tor ant so bad... really I think....never tried Tor hehe

myopic1 · Jan 14, 2015

777lady said:
I try this also, but same effect.

I really don't know why that wouldn't work for you but I would put good money on it not being the fault of the requests library...which is a pretty awesome library. My advice would be to A) Check your proxies actually work B) Check what sort of authentication your proxies use (if any) and make sure you're providing it C) Use a different library such as urllib2 or something similar.

777lady · Jan 14, 2015

myopic1 said:
I really don't know why that wouldn't work for you but I would put good money on it not being the fault of the requests library...which is a pretty awesome library. My advice would be to A) Check your proxies actually work B) Check what sort of authentication your proxies use (if any) and make sure you're providing it C) Use a different library such as urllib2 or something similar.

A and B I checked, also I use dedicated private proxies. I think I will try C now maybe this can solve the problem. Will tell you if it than works.

python crawler

777lady

Newbie

bartosimpsonio

Elite Member

777lady

Newbie

myopic1

Regular Member

777lady

Newbie

TheVegan

Junior Member

tony_d

Elite Member

seoproranker

Newbie

myopic1

Regular Member

777lady

Newbie

Main Menu

Marketplace

Making Money

BlackHat World