Selenium vs Playwright - For Web automation and scraping

[ATuringtest]

Also @Captaintj don't discount already make solutions, for example @rootjazz makes software for automating Instagram and Twatter, they both have built in scrapers that have shit tons of options.

 

[xelex]

Very interesting thread, with some quite knowledgeable people in here. I've managed to apply some hacks to the selenium chromedriver to make some Google Captchas easier, but that's where my expertise ends on this topic...

Autohotkey I will guarantee you %100, if u spend the time to learn an actual TSL language no one on this planet will ever detect what your doing.
But here's the problem with people - they are not willing to put the effort in now to reap the rewards later. They just want quick fixes to complex solutions, a fast turn around, easy money.

This seems like a bulletproof idea, could even get a plain old java bot class to do the input for you.
But how do you actually go about interacting with a page, i.e. reading the content and figuring out where to click?
Own, custom browser plugin?

 

[ATuringtest]

Very interesting thread, with some quite knowledgeable people in here. I've managed to apply some hacks to the selenium chromedriver to make some Google Captchas easier, but that's where my expertise ends on this topic...

This seems like a bulletproof idea. but how do you actually go about interacting with a page, i.e. reading the content and figuring out where to click?
Own, custom browser plugin?

Ok, this is a big topic.
If you wanted to run say your Class A SM accounts, the money earners etc, then I use AHK in a Virtual environment + 3g rotating proxies.
I operate one account at time, on several virtual machines.

Back to your question : Where to click ?
Easy, you use either OCR recognition or Image recognition.
I slung together a quick demo in like 10 mins for you.

 

[xelex]

Ok, this is a big topic.
If you wanted to run say your Class A SM accounts, the money earners etc, then I use AHK in a Virtual environment + 3g rotating proxies.
I operate one account at time, on several virtual machines.

Back to your question : Where to click ?
Easy, you use either OCR recognition or Image recognition.
I slung together a quick demo in like 10 mins for you.

Very nice, I actually had started a poker botting project quite a few years back, and I had been experimenting with that set up. Image recognition, OCR, AHK.

Doesn't sound too easy though. Perhaps for repetitive action based tasks, where you're just filling out the same form and pressing buttons, it shouldn't be too difficult, just load up some target images, and fill out forms accordingly.

But what if we need to scrape info from the pages? Or we have a lot of forms and don't want to spend a lot of time fiddling with images? What about pixel discrepancies that could cause problems with image recognition?

 

[ATuringtest]

Very nice, I actually had started a poker botting project quite a few years back, and I had been experimenting with that set up. Image recognition, OCR, AHK.

Doesn't sound too easy though. Perhaps for repetitive action based tasks, where you're just filling out the same form and pressing buttons, it shouldn't be too difficult, just load up some target images, and fill out forms accordingly.

But what if we need to scrape info from the pages? Or we have a lot of forms and don't want to spend a lot of time fiddling with images? What about pixel discrepancies that could cause problems with image recognition?

If your scraping you go back to GET + Selenium
For forms you only set them up once and then run them if say you have 100 things a day to do every day then set up once takes time agreed - but its about not being detected on SM etc that AHK shines at.

 

[Shopon Ahmed]

Im am using selenium without facing any troubles.

 

[crainichum]

I will recommend selenium as is very easy to set up and do the job with it. No matter what bots you use if your activity don't look natural will easy be banned

 

[iLogin]

Selenium is the best

 

[IDENTORY]

If you have to face serious anti-bot systems, it is better to choose Puppeteer/Playwright rather than Selenium. Selenium is almost impossible to hide from serious anti-bot systems, while Puppeteer/Playwright use more modern ways of controlling the browser that can be hidden.

 

[brucebanner]

I was using selenium times ago and yes you can do almost anything but once you face real anti-bot detection selenium will be flagged. I found puppeteer more lowkey and faster.

-U cant scale with selenium, too unstable
-U cant perfectly hide with selenium

Better use puppeteer mixed with httprequest. Only my 2 cents!!!

 

[stefcaam]

I wish to make bots that scrape data from one social media and upload it to another. But I am not sure which is the best to avoid bans.
Selenium or Playwright. I do not have much experience in either.

Selenium.

But you can not avoid bans, It's You vs Big Development Team? Who do you think wins?

If their webautomation protection system is good, you have very little chance.

However, you can take a break between postings for like 10 seconds ?? (do not to post 5 posts at once)...

 

[SliverHawk]

If you are using user-agent spoofing and xhr the detection probability is close to none!

The idea is very interesting. I will give it a try. Perhaps XHR (XML HttpRequest) is the key? We can block the request, then modify and send it, right?

 

[batmax]

as other said..... there are billions $ spent for bots detection.

I tried selenium but I'm playing with "playwright" for few months and IM really satisfied.

U dont need a chromedriver bur u still have the webdriver flag On if u dont hide it in the same way you do in selenium.

Dont remember who said sad but the real question is not if the SM/website block or not automated browser, but when/why ur bot account has been banned.

Some website detect u as bot directly and block you
others detect u as bot but dont block you at login byt only after shitty actions (behaviors)
IP reputation also is very important !!


btw AHK is great !!!! u can find a "record/replay" script easily but its only for windows i guess, there is SIKULI for unix.

 

[SliverHawk]

Haha, we will definitely be caught if they focus on us. So, we just need to try our best not to be targeted by them.

After a series of experiments, I found that the best approach is to intervene and automate the process by modifying the http request, although it is quite challenging. However, in return, the accuracy is extremely high.

 

[Nikolaos]

Playwright is my favourite so i would probably go with this.

 

[juggnfinesse]

I used Puppeteer in the past and can recommend.

 

[4440]

I use Pullovers Macro Creator everyday, in fact he's done a few pod cast with selenium guys showing how to inject JavaScript into chrome tools to automate it without being detected.. it much harder than selenium to do becasue it pure scripting but once you have a bit of time using it under your belt .. works great

hi can you share the link of the said podcasts or the name of the podcast? I tried searching but got no results, thanks!

 

[OccultKingdoms]

I use Pullovers Macro Creator everyday, in fact he's done a few pod cast with selenium guys showing how to inject JavaScript into chrome tools to automate it without being detected.. it much harder than selenium to do becasue it pure scripting but once you have a bit of time using it under your belt .. works great

I know anything. Boookkmarked

 

[ATuringtest]

hi can you share the link of the said podcasts or the name of the podcast? I tried searching but got no results, thanks!