My Web scraping code stop working because of...

Zhen

Zhen

Registered Member
Joined
Mar 22, 2024
Messages
70
Reaction score
22
helloo forum,
hope my post finds you well
Recently I made a web scraping codeusing selenium base and Bs to scrape users email from a well known website. first, it works perfectly. but he stopped on the claud flare captcha page then, I try to do it manually and couldn't skipped it tho.
If any one here have a good knowledge in selenium I would be greatful if he fix that bug.
 
Advertise on BHW
If you click the button and the page reloads again on Cloud Flare. Maybe the issue is in the fingerprint, try to configure it, also connect the proxy. As for the check button, you can click on it, finding its xpath. If you can't fix it, write to me, I will help you
 
Yes if you're using Selenium then you need to use an undetectable version of selenium, there are plenty of tutorials on YouTube for it
 
Cloudflare captchas usually mean the site is blocking automation, so the best approach is to review the site’s terms and switch to official APIs or adjust your scraper to respect limits instead of trying to bypass protections.
 
I'm probably going to disappoint you, but even if you manage to bypass Cloudflare, it'll only be a temporary solution. They're making great progress and doing everything they can to keep automated bots out of their sites.
 
Webscraping is fucking hard. It's not 2010 any more where it's just a simple scroll, find data, scrape it, interact with pagination and buttons. These days 90% of the time you're fighting obfuscated javascript, shadow DOMs and captcha.
 
You need to use anti detect browsers or spoof your fingerprints very well to be able to do scraping these days mate. And for the captcha you can automate solving that easily.
 
CF can probably detect selenium

You need to make your scraper stealth
 
Cloudflare probably fingerprinted your browser/session already.

Selenium alone is weak now, most people combine rotating residential proxies, browser profiles and lower scrape rates to keep sessions alive longer.
 
I've spent more time making scrapers look human than actually extracting the data.... If your scraper suddenly stopped working after running fine for a while, that's usually a sign the site changed its anti-bot detection, not your code breaking.
 
you can use captcha solver but first you need to add proxy support to your code then add fingerprint profile generator to change fingerprint and user agent and others for each session
that will fix the issue
 
Selenium gets caught by Cloudflare's bot heuristics fast — it's the navigator.webdriver flag, the missing chrome.runtime properties, and the WebGL fingerprint that gives it away, not your scraping logic. Two paths: keep Selenium and patch it with undetected-chromedriver (replaces the patched binary that doesn't set webdriver flag), or switch to a CF-aware lib — cloudscraper for simple JS challenges, FlareSolverr if you need a proxy service that solves the full challenge headless and gives you back cookies. If even manual solving doesn't pass, the site likely escalated you to Turnstile + IP scoring — at that point you need to rotate to a clean residential IP before any solver works.
 
standard selenium is dead for cloudflare websites bro, manual bypass wont work either cuz ur browser fingerprint is already leaked as a bot.
 
are you using proxies? This is what they are there for.
 
if it broke suddenly and you're seeing a cloudflare page, that's probably turnstile, not a normal captcha — and it's not a "solve the puzzle" thing, it's checking your browser fingerprint and tls handshake. plain selenium gets flagged on the headless fingerprint before the challenge even renders. two separate fixes: (1) the fingerprint — undetected-chromedriver or a real browser profile so you're not screaming automation, and (2) the token itself, which a solver returns (cf-turnstile-response) for you to inject. doing only one won't cut it. and don't touch proxies until the fingerprint's sorted — that's not what's biting you here.
 
Selenium is dead, same as undetected-chromedriver. There are better libraries to work with, that don't get detected so easily.
 
u won't fix it just by replacing Selenium

u should learn more tools for that to avoid CF and other captcha,

if u wanna i got some solutions dm me if u ready for deal
 
This isn't really a selenium bug, it's the site telling you that session/traffic isn't welcome. If even manual solve doesn't pass, changing xpath or adding waits wont do much because the browser/IP is already scored bad. For email scraping especially, better use sources you're allowed to pull from or an API/export if they have one, otherwise you'll just keep fixing the same thing every few days when CF changes again.
 
Of course it is Cloudflare is built precisely to stop scraping via automation, which means that Selenium and BS4 will no longer be enough to bypass Cloudflare.
Instead of attempting to fix such a problem, you will have to use either an API provided by the company itself or a permission-based data source
 
I think it is because of Cloudfare flagged your scraper react like automated behavior and browser fingerprint like a bot. Also your IP address also one of the reason, Cloudfare from a known IP address without rotation. Of course may have other reasons that I unforseen..
To solve it, I think you may consider switch to Playwright, it may reduce Cloudflare flag. You also may consider use a premium proxy service that have dynamic IP address all time.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top