Python Requests, Need help. Getting Rate Limited With 70 Proxies???

while 1:
timer = datetime.now()
print(f'new check {timer}')
while 1:
time.sleep(0.5)
resp = requests.get(URL, headers=headers, proxies=proxies)
if resp.status_code == 200:
proxy = next(proxy_cycle)
proxies = {
"http://": proxy,
"https://"roxy
}
break
else:
print(resp.status_code)
proxy = next(proxy_cycle)
proxies = {
"http://": proxy,
"https://"roxy
}
print(proxy)
###### continues the loop with the rest of the code

Click to expand...

proxy = next(proxy_cycle)
proxies = {
"http://": proxy,
"https://"roxy
}

Click to expand...

timer = datetime.now()
print(f'new check {timer}')

Click to expand...

Hoichi said:

I recommend you randomly choosing proxy from the list. If it will not work, you should change proxy provider or maybe google is banning your pool of proxy. Remember that proxy provider is not rotating proxy every minute, usually it is done in 24h.

Click to expand...

spectrejoe said:

Can someone give me a hand? I'm using 70 proxies for an API that allows you to do 1 request per second.



If I go to the URL manually I can spam F5 F5 F5 F5 like 10x in a row and I don't get limited but my proxies are getting error 429 too many requests like wtf? I'm using:



Even on successful attempts to go to the next proxy.
I track the run time with



And it's roughly 0.7s with time.sleep(0.5) on each loop. So for each proxy It should be doing AT MOST 1 request every (0.7*70=49 seconds)???? Yet they're still often getting rate limited....

Click to expand...

ANewBeginning said:

It's possible that you aregetting blocked at a proxy level, or even the datacenter where your proxies are from.

Change to residential rotating proxies that change with every request, so you get different datacenters, info,, etc. They are pricey but it's a good way to bypass those filters

Click to expand...

spectrejoe said:

Is that possible? That they change with every request? Considering I'm making a request every 0.1 seconds

Click to expand...

TheOverlord said:

not any rotating proxy does this, but backconnect proxies change with each request. Datacenter proxies are highly spammed and residential proxies+ are expensive, that is the main tradeoff

Click to expand...

spectrejoe said:

oof yea extremely expensive from what I'm seeing.

Do you know any good provider for "normal" proxies? Because I feel like my original IP is most likely leaking or getting detected by cloudflare somehow.

Click to expand...

GoGuerilla said:

Call this at the same time to ensure you are getting the right/different IP.

https://api.ipify.org/?format=json

Click to expand...

spectrejoe said:

Found out the issue, the proxies weren't being used through the requests.get it's working fine now, although a tiny bit slow due to proxies taking a bit to connect, I guess the key here is getting super fast proxies. if there's any recommendation for decent fast proxies lmk

Click to expand...

spectrejoe said:

Found out the issue, the proxies weren't being used through the requests.get it's working fine now, although a tiny bit slow due to proxies taking a bit to connect, I guess the key here is getting super fast proxies. if there's any recommendation for decent fast proxies lmk

Click to expand...