Scraping automation

SicariusDread · Aug 26, 2023

Which is you preferred tool or library for scraping and submission automation?

I'm looking to check options to scrape the most difficult places to be scraped (the ones that check for human behavior mostly, like the ones protected by Cloudflare)

So far I've tried Selenium and Puppeteer without great results. Maybe it's my skills with this two tools, but maybe there are more sophisticated options that can improve the way to pass through.

Hiexperts · Aug 29, 2023

You can use Scrapy, Playwright, Splash, and Crawlera.

Somethingidktbh · Aug 29, 2023

SicariusDread said:
Which is you preferred tool or library for scraping and submission automation?

I'm looking to check options to scrape the most difficult places to be scraped (the ones that check for human behavior mostly, like the ones protected by Cloudflare)

So far I've tried Selenium and Puppeteer without great results. Maybe it's my skills with this two tools, but maybe there are more sophisticated options that can improve the way to pass through.

For advanced scraping that bypasses anti-scraping measures like Cloudflare, you might consider using Scrapy with CrawlERA, Splash for JavaScript rendering, Microsoft's Playwright for more sophisticated interactions, Residential Proxies for less detectable IP addresses, Capybara for user-like interactions, or Octoparse for a GUI-based approach.

SicariusDread · Aug 29, 2023

Somethingidktbh said:
Capybara for user-like interactions, or Octoparse for a GUI-based approach.

Good amount of software, all at once? For example Capybara and Playwrite are not mutually exclusive?

good_luck_commander · Aug 29, 2023

I have used the python selenium library and was very happy with that. If you know how to code in python. Chatgpt can pretty much just generate most of the code for you though, so most basic python skills would be enough.

Voordeel · Sep 4, 2023

My preferred tool for scraping and submission automation in challenging scenarios, such as websites protected by Cloudflare, would be Scrapy with the use of middleware like Crawlera or rotating proxies. This combination offers better control and scalability for handling difficult-to-scrape websites.

seancoder · Sep 4, 2023

Somethingidktbh said:
For advanced scraping that bypasses anti-scraping measures like Cloudflare, you might consider using Scrapy with CrawlERA, Splash for JavaScript rendering, Microsoft's Playwright for more sophisticated interactions, Residential Proxies for less detectable IP addresses, Capybara for user-like interactions, or Octoparse for a GUI-based approach.

What are the speed of these tools compared to a something like selenium ?

Scraping automation

SicariusDread

Junior Member

Hiexperts

Power Member

Somethingidktbh

Newbie

SicariusDread

Junior Member

good_luck_commander

Junior Member

Voordeel

Banned-Rules Violation

seancoder

Failed to resolve a dispute resolution thread.

Main Menu

Marketplace

Making Money

BlackHat World