- Jul 22, 2023
- 194
- 36
Heads up if your rotation looks broken - 90% of "the proxy isn't rotating!" issues we see are actually HTTP keep-alive on the client side.
If your scraper opens a persistent HTTP/1.1 connection and sends 100 requests through it, the proxy assigns ONE IP for that tunnel and reuses it. Doesn't matter that you're set to "rotating" - keep-alive means one TCP socket, one upstream session, one exit IP. You'll burn 100 requests on the same IP without realizing.
Quick fixes depending on your language:
for i in $(seq 1 10); do<br> curl --no-keepalive -x "http://USER:[email protected]:12286" http://ip-api.com/json | jq -r .query<br>done<br>
If you're getting fewer than ~8 unique IPs, keep-alive is the culprit, not the pool.
If your scraper opens a persistent HTTP/1.1 connection and sends 100 requests through it, the proxy assigns ONE IP for that tunnel and reuses it. Doesn't matter that you're set to "rotating" - keep-alive means one TCP socket, one upstream session, one exit IP. You'll burn 100 requests on the same IP without realizing.
Quick fixes depending on your language:
- Python requests: don't reuse a Session() object - either call requests.get() each time, or session.close() between calls.
- Python httpx: pass httpx.Client(http2=False) and avoid context-manager reuse if you want per-request rotation.
- Node axios / fetch: pass an agent with keepAlive: false, or new http.Agent() per call.
- Curl: --no-keepalive flag.
- Go net/http: set Transport.DisableKeepAlives = true on the client.
for i in $(seq 1 10); do<br> curl --no-keepalive -x "http://USER:[email protected]:12286" http://ip-api.com/json | jq -r .query<br>done<br>
If you're getting fewer than ~8 unique IPs, keep-alive is the culprit, not the pool.
