Scrape for domains

Chimmy1990

Junior Member
Joined
Mar 24, 2018
Messages
199
Reaction score
158
I’ve found a website that lists several sites I’m interested in. However, each listing uses an affiliate link/button that redirects to a blog page, where the actual URL is revealed.
 
Probably "paid" promotion, so they would be able to track the clicks,
Or just cloaking the redirects or something similar
 
So? (Let somebody) Code something in python to process the data you want.
 
I think you should try Playwright. It spins up a headless browser, runs all the client-side stuff, and gives you the final URL after everything’s loaded. If you’re doing this at scale def go with the async version and rotate proxies so you don’t get flagged. But if the redirect is just a simple meta tag or something static, you could run with a lighter setup like requests plus BeautifulSoup
Sure. I'll give it a try. Thanks
 
Probably "paid" promotion, so they would be able to track the clicks,
Or just cloaking the redirects or something similar
Oh, I see. It makes it quite hard to scrape all 5,000 domains because I have to click on each one individually to get the actual URL.
 
Back
Top