Simple Python Scraper Template with Proxy Rotation & Anti-Ban Delays

Hello BHW!
I've been browsing the forum for a while and noticed many guys here struggle with basic scraping blocks and proxy management. I wanted to share a clean, lightweight Python boilerplate that I use for my automation tasks.
It handles random User-Agents, rotating proxies from a list, and includes random backoff delays to prevent immediate bans.

import requests
import random
import time
from bs4 import BeautifulSoup

# List of proxies to rotate (format: 'iport' or 'userass@iport')
PROXY_LIST = [
"http://your_proxy1ort",
"http://your_proxy2ort",
"http://your_proxy3ort"
]

# List of common User-Agents to mimic different browsers
USER_AGENTS = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2.1 Safari/605.1.15",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"
]

def get_secure_session():
session = requests.Session()

# Set random User-Agent
session.headers.update({"User-Agent": random.choice(USER_AGENTS)})

# Set random Proxy if list is not empty
if PROXY_LIST:
proxy = random.choice(PROXY_LIST)
session.proxies = {"http": proxy, "https": proxy}

return session

def fetch_data(url):
session = get_secure_session()
try:
# Added timeout to prevent script from hanging
response = session.get(url, timeout=10)
if response.status_code == 200:
return response.text
else:
print(f"[-] Failed with status code: {response.status_code}")
return None
except requests.exceptions.RequestException as e:
print(f"[-] Connection error: {e}")
return None

# Example usage
if __name__ == "__main__":
target_url = "https://httpbin.org/ip" # Use this to test proxy rotation

print("[+] Starting request...")
html_content = fetch_data(target_url)

if html_content:
print("[+] Success! Parsing data...")
# You can add your BeautifulSoup logic here

# Random delay between 2 and 5 seconds to stay under the radar
time.sleep(random.uniform(2.5, 5.5))

Features included:
Simple HTTP session management using requests.
Randomized headers to avoid fingerprinting.
Basic try/except block to catch bad or dead proxies without crashing the script.
Feel free to copy, modify, and use it for your web scraping or data extraction tasks. Let me know if you need help adjusting this for dynamic JavaScript websites (Playwright/Selenium) — I might drop a guide on that later if you guys are interested!

Thanks for sharing this @DevNexi. One quick thing though... since you call get_secure_session inside the fetch function, wouldn't that spin up a brand new session on every single request? If someone loops this over a few hundred pages, it kinda defeats the main benefits of requests.Session() like connection pooling and keeping cookies.

Still a clean starter template for basic HTML sites, but these days almost everything has Cloudflare or Akamai anyway. Usually have to move to something like tls-client or playright pretty quick to avoid getting blocked instantly. Would definitely be interested in that Playwright guide if you write it up.

Good starter for basic stuff. @PPCPirate already nailed the session issue, thats the main thing i'd fix... creating a new session every call kills the whole point of pooling and you burn a proxy per request instead of sticking to one for a batch.

Other thing, the sleep at the very end only runs once after the whole script finishes, so its not actually throttling anything between requests. If you're looping over pages you want the backoff inside the loop.

And yeah httpbin is fine for testing rotation but the second you hit anything real with cloudflare plain requests gets walled instantly no matter how nice your UA list is. curl_cffi or tls-client will get you a lot further before you even need to touch playwright. Would still read the playwright writeup though.

Yeah, and I’d probably add a small cooldown for failed proxies too.

With random.choice() the same dead IP can get picked again almost immediately. Even a basic fail count and temporary skip would save a lot of pointless retries

rotating proxies locally from a flat list is always a headache because of the dead IPs. unless you are using high quality private proxies, it is usually way easier to just use a backconnect proxy where the provider handles the rotation and dead IPs on their end. saves you writing a ton of retry logic...

if you do stick to a local list though, a quick and dirty way is to just use list.remove(proxy) in the except block. not perfect but keeps the script moving without hitting the same dead wall over and over.