Selenium vs Playwright - For Web automation and scraping

[Toz]

C'mon guys, let's save the elitism for those certain online places full of dicks. Great insight being shared on here, but let's dial it down a bit so we don't overflow the thread with insults and soap boxes. And yes, pun intended.

 

[Mactronn]

right here's the facts..

Selenium was designed for Web Automation, it came originally from TSL (Test Script Language). Google 100% knows your using it.
Do they care ?
No
Do SM sites know your using it ?
Yes
Do they care?
Some do - Some don't
Vast - Vast majority of SM bots use backend API's or stream injectors.
All those that say.. "I use selenium on my X site and I've never been banned", just wait, When those in charge find their girlfriend giving a blowjob to the post man, then your SM accounts will be flushed down the shitter, like his dignity.

This can help Undetectable Chrome Project I use it sometimes... but for scraping and nothing else.

What do I use ?

Autohotkey I will guarantee you %100, if u spend the time to learn an actual TSL language no one on this planet will ever detect what your doing.
But here's the problem with people - they are not willing to put the effort in now to reap the rewards later. They just want quick fixes to complex solutions, a fast turn around, easy money.

If you are using user-agent spoofing and xhr the detection probability is close to none!

 

[lucky.sparks]

C'mon guys, let's save the elitism for those certain online places full of dicks. Great insight being shared on here, but let's dial it down a bit so we don't overflow the thread with insults and soap boxes. And yes, pun intended.

I don't think there is anything to do with elitism - if people trash the thread with advice or comments like this below which is simply untrue, then what value to the thread there is? People who want to learn will read and if the thread is full of things that are untrue but served with confidence, it just gets hard to distinguish good information from bad. I know this because I learned marketing from this forum(and still do). Only by calling them out, there is a chance to add value. In other words, sometimes you need to be a dick, for sake of others.

If you are using user-agent spoofing and xhr the detection probability is close to none!

 

[madacot233]

If you are using user-agent spoofing and xhr the detection probability is close to none!

https://en.wikipedia.org/wiki/TCP/IP_stack_fingerprinting

 

[JavascriptPufferfish]

right here's the facts..

Selenium was designed for Web Automation, it came originally from TSL (Test Script Language). Google 100% knows your using it.
Do they care ?
No
Do SM sites know your using it ?
Yes
Do they care?
Some do - Some don't
Vast - Vast majority of SM bots use backend API's or stream injectors.
All those that say.. "I use selenium on my X site and I've never been banned", just wait, When those in charge find their girlfriend giving a blowjob to the post man, then your SM accounts will be flushed down the shitter, like his dignity.

This can help Undetectable Chrome Project I use it sometimes... but for scraping and nothing else.

What do I use ?

Autohotkey I will guarantee you %100, if u spend the time to learn an actual TSL language no one on this planet will ever detect what your doing.
But here's the problem with people - they are not willing to put the effort in now to reap the rewards later. They just want quick fixes to complex solutions, a fast turn around, easy money.

cool to see someone mention AHK in the wild. i learned a bit of autohotkey back in the day and the community is really helpful. lots of good tools built for it. there was even the Pulover's Macro Creator,never got it to work but it looked great

https://www.autohotkey.com/boards/viewtopic.php?style=7&t=143&start=880​

 

[ATuringtest]

cool to see someone mention AHK in the wild. i learned a bit of autohotkey back in the day and the community is really helpful. lots of good tools built for it. there was even the Pulover's Macro Creator,never got it to work but it looked great

https://www.autohotkey.com/boards/viewtopic.php?style=7&t=143&start=880​

I use Pullovers Macro Creator everyday, in fact he's done a few pod cast with selenium guys showing how to inject JavaScript into chrome tools to automate it without being detected.. it much harder than selenium to do becasue it pure scripting but once you have a bit of time using it under your belt .. works great

 

[Gogol]

I wish to make bots that scrape data from one social media and upload it to another. But I am not sure which is the best to avoid bans.
Selenium or Playwright. I do not have much experience in either.

It really depends. Just choose the one you like to begin with. Then start comparing. Which one gets you further?

 

[JavascriptPufferfish]

I use Pullovers Macro Creator everyday, in fact he's done a few pod cast with selenium guys showing how to inject JavaScript into chrome tools to automate it without being detected.. it much harder than selenium to do becasue it pure scripting but once you have a bit of time using it under your belt .. works great

to be fair with pullover, i was very impatient back then and skimmed through the tutorials, so its 100% my fault i didn't get it to work. i am learning JS right now, but as soon as i get to a level i am satisfied with, i will take on AHK again and check that out. i remember buying Joe Glines courses, good stuff. it always surprises me how little is AHK talked about here, considering its free, well documented and has such a good community and tools.

 

[Mactronn]

https://en.wikipedia.org/wiki/TCP/IP_stack_fingerprinting

Looks like you have never heard of selenium-stealth, fake-useragent libraries..

 

[lucky.sparks]

Looks like you have never heard of selenium-stealth, fake-useragent libraries..

the guy you just called out is one of the most skilled members on this forum regarding automation and building bots.

either you did not even tried to understand whats written under that wiki link or you are trolling, either way - selenium stealth does not help much against advanced anti-bot detection companies, same with user agent changing, running a real stealth automation project requires a lot more than just these simple and basic things. there are literally dozens of multi million dollar companies whos only task is to detect and prevent using selenium on their client websites.

 

[The Economist]

So I usually do this stuff on a deeper level using requests from the terminal and reading the response and extracting what I need and customizing every script according to each situation...I don't like selenium tbh and plus I like using my low specs computer so it's slow running another automated browser in which you have limited scope to mask your fingerprints...granted some websites employ many token generation techniques to attach to each request but I find my way around them most of the time....using the web interface is easier however when I tried reverse engineering some apps...I'll usually find myself needing to decompile the app to find what functions creates what token that goes along the selected request....

Bottom line...I'm a libertarian...I prefer something that gives more maneuver margin....
ofc I'm not expert...I'm just following my pragmatism to get to what actually brings results...

 

[noellarkin]

So I usually do this stuff on a deeper level using requests from the terminal and reading the response and extracting what I need and customizing every script according to each situation...I don't like selenium tbh and plus I like using my low specs computer so it's slow running another automated browser in which you have limited scope to mask your fingerprints...granted some websites employ many token generation techniques to attach to each request but I find my way around them most of the time....using the web interface is easier however when I tried reverse engineering some apps...I'll usually find myself needing to decompile the app to find what functions creates what token that goes along the selected request....

Bottom line...I'm a libertarian...I prefer something that gives more maneuver margin....
ofc I'm not expert...I'm just following my pragmatism to get to what actually brings results...

I keep attempting to shift from browser automation to HTTP requests, but it gets so damnably hard, for some of these sites every button click generates a ton of requests on my Fiddler monitor. Eventually I just end up going back to browser automation after a few days lol
How much time did it take you to get comfortable with working just with HTTP requests? The only thing I'm comfortable using requests with is when I'm working with an API like Text2Data or OpenAI.

 

[The Economist]

I keep attempting to shift from browser automation to HTTP requests, but it gets so damnably hard, for some of these sites every button click generates a ton of requests on my Fiddler monitor. Eventually I just end up going back to browser automation after a few days lol
How much time did it take you to get comfortable with working just with HTTP requests? The only thing I'm comfortable using requests with is when I'm working with an API like Text2Data or OpenAI.

Honestly...it's tricky as you said to simulate the HTTP requests...however when using APIs it's pretty straightforward as long as you follow the documentation....you're not supposed to recreate these requests you see in Fiddler or the browser sev tools so don't expect it to be as clear as an API docs...what I would say is maybe to MICRO analyze every browser action and go step by step...

 

[Toz]

if people trash the thread with advice or comments like this below which is simply untrue, then what value to the thread there is?

Lots. For example, as someone who uses Selenium in both personal and professional capacities, I've genuinely never heard of Playwright. This thread allowed me to learn about something I hadn't known before. As for any false and / or misleading information, I understand what you're saying. It's important to keep things accurate.

 

[ATuringtest]

Lots. For example, as someone who uses Selenium in both personal and professional capacities, I've genuinely never heard of Playwright. This thread allowed me to learn about something I hadn't known before. As for any false and / or misleading information, I understand what you're saying. It's important to keep things accurate.

There's Puppeteer as well,

Most things that you can do manually in the browser can be done using Puppeteer! Here are a few examples to get you started:

• Generate screenshots and PDFs of pages.
• Crawl a SPA (Single-Page Application) and generate pre-rendered content (i.e. "SSR" (Server-Side Rendering)).
• Automate form submission, UI testing, keyboard input, etc.
• Create an up-to-date, automated testing environment. Run your tests directly in the latest version of Chrome using the latest JavaScript and browser features.
• Capture a timeline trace of your site to help diagnose performance issues.
• Test Chrome Extensions.

and Microsoft has a macro system for the system that targets COM's, including Edge.

 

[Mactronn]

the guy you just called out is one of the most skilled members on this forum regarding automation and building bots.

either you did not even tried to understand whats written under that wiki link or you are trolling, either way - selenium stealth does not help much against advanced anti-bot detection companies, same with user agent changing, running a real stealth automation project requires a lot more than just these simple and basic things. there are literally dozens of multi million dollar companies whos only task is to detect and prevent using selenium on their client websites.

So, you are not smart enough to make an undetectable bot. May I ask how we can run hundreds of social media accounts without any problem? We developed tons of pretesting tools in Selenium so I really know what I'm talking about. But it's not possible.. if you say so.

 

[lucky.sparks]

So, you are not smart enough to make an undetectable bot. May I ask how we can run hundreds of social media accounts without any problem? We developed tons of pretesting tools in Selenium so I really know what I'm talking about. But it's not possible.. if you say so.

I did not said its not possible, never did that, please quote me on that then.

There are literally thousands of ways how social network can detect you are doing something fishy.

You said that it's enough to just use xhr and user-agents, which is simply not true because some networks will require fingerprints to be passed along with those xhr requests or requests to be signed, for example - snapchat or tiktok is doing this, while tiktok is still somewhat doable, then automating snapchat API is extremely hard.

Then you said selenium-stealth with useragent libraries is way to go, which is partly not true either because:
a) you cant emulate any user agent because of differences in TLS fingerprints, that is what @madacot233 tried to tell here. Even more, there is even no need to use such useragent libs, all you must do is to use the most used user-agents for the same OS you run your automation from.

b) some anti-bot vendors like shape security will use backend solutions to check your requests and will easily detect differences even in smallest things - like if all headers are present and even if order in them matches the useragent. as per my knowledge, no publicly available selenium stealth/patched version does not solve this, if you have developed your own inhouse version, cheers to you but it does not help or contribute to this thread and the main question.

And if your developed solution works for your specific social network, it does not mean it's bulletproof or can be applied for every use case, they are all different.

OP was asking about the generic use cases and till now only @ATuringtest has provided a good solution/answer.

 

[OccultKingdoms]

What do I use ?

https://www.autohotkey.com/ I will guarantee you %100, if u spend the time to learn an actual TSL language no one on this planet will ever detect what your doing.
But here's the problem with people - they are not willing to put the effort in now to reap the rewards later. They just want quick fixes to complex solutions, a fast turn around, easy money.

Take it easy. That's great advice you've posted on here.
As you and I pointed out one day - Selenium alone doesn't fit botting tactics.
It's the wrong tool for that purpose and is a mismatch.

I have used Autohotkey recently to write faster and complete one of those tasks you want to sweep under the rug and forget forever.
After violating its true purpose and underestimating its potential, I arrived here and realized my time wasn't wasted on picking up on that excellent piece of software.
Thanks for sharing this hint.

 

[Mactronn]

I did not said its not possible, never did that, please quote me on that then.

There are literally thousands of ways how social network can detect you are doing something fishy.

You said that it's enough to just use xhr and user-agents, which is simply not true because some networks will require fingerprints to be passed along with those xhr requests or requests to be signed, for example - snapchat or tiktok is doing this, while tiktok is still somewhat doable, then automating snapchat API is extremely hard.

Then you said selenium-stealth with useragent libraries is way to go, which is partly not true either because:
a) you cant emulate any user agent because of differences in TLS fingerprints, that is what @madacot233 tried to tell here. Even more, there is even no need to use such useragent libs, all you must do is to use the most used user-agents for the same OS you run your automation from.

b) some anti-bot vendors like shape security will use backend solutions to check your requests and will easily detect differences even in smallest things - like if all headers are present and even if order in them matches the useragent. as per my knowledge, no publicly available selenium stealth/patched version does not solve this, if you have developed your own inhouse version, cheers to you but it does not help or contribute to this thread and the main question.

And if your developed solution works for your specific social network, it does not mean it's bulletproof or can be applied for every use case, they are all different.

OP was asking about the generic use cases and till now only @ATuringtest has provided a good solution/answer.

I provided libraries just for references. This is not development thread, so further discussion is not needed. The OP may digg deeper if interested.

As I said in my first post, detection is close to none. I stand with that statement.

 

[ATuringtest]

I've always thought about doing a Udemy course.. "Making PVA Gmail Accounts with AHK Fully Automated". Might get around to it one day just to shove stick up Googles ass.