How should I get started if I want to learn browser automation?

[karupoiss]

Hi!

I have a bit of a programming background - I have bachelors degree in informational technology but the bad thing is that I learned it 10 years ago and don't remember much of it (we learned java, python and php a bit).

I want to start making my own bots for stuff like social media automatization. Things like sending PM-s with multiple accounts at the same time, scraping content, posting, upvoting etc. I have read there are thousands of things you need to fake about browsers such as canvas and IP and user id etc.. I have no idea how or from where to start?

Any suggestions?

 

[masta2133]

Custom bots:
Use: Selenium + python. Look for the book automate boring stuff with python, you'll learn a ton. You should learn python anyways if you plan on working on bots.
Use: Cheerio + nodejs (javascript). I find it more intuitive but you gotta learn javascript. You can also control the browser through Javascript.
Use: C# for more advanced bots, this allows you to have more control and to build interfaces as well.

Already available software:
Otherwise, you can simply rely on Zennoposter, it is good enough and helps you get started quicker. But there is a learning curve in that too.
The other advantage is that you can buy custom made templates for cheap. There also lots of free templates to learn from.
And more importantly, it is as intuitive as it gets. I usually refrain from using that kind of software but Zennoposter's UI design makes it easy to use and learn.
It is also highly customizable as you can include custom C# code snippets if you need granular control over the tasks that need to be accomplished.

P.S.: Sorry about the thousand edits

 

[karupoiss]

I would use Zennoposter but it seems very primitive as social networks have all those browser fingerprinting stuff.. how could I learn more about the browser fingerprinting etc to avoid bans?

 

[kingvestor]

I use ubot studio for some simple stuff, as you said you need more than ubot or zenposter these days

 

[NumUno]

Or you can consider Winautomation... although I think that they were bought by some big guns recently... not sure....

 

[Meerakat]

The best and easiest way to get started with automation is to first be familiar with how automation works.

For example, you need to have a bit of knowledge with using elements since when automating them you need to either:

1. Input text
2. Click a button
3. Or god knows whatever else

You will know what I am talking about once you go in-depth with how web automation works.

I suggest you go for puppeteer. Puppeteer as a module is very easy to use (it has a lot of documentation available, plugins, etc.. to help you out with automation.)

=> puppeteer - npm (npmjs.com)

 

[ziko12345]

Although i don't have any experience myself, i heard a bunch of people using selenium.

 

[ThePacketeer]

I mainly use puppeteer in nodejs (javascript) https://pptr.dev/
This combined with puppeteer-extra-stealth plugin to emulate unique users and avoid bot detection https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth
Then use ali's anyproxy to tunnel to a residential proxy and intercept ssl requests if needed

 

[TasDePixels]

The problem with Puppeteer, nightmare.js, and other libraries that rely on full browser emulation is that, if you screw up the implementation process, your bot will inevitably consume a fucktone amount of resources. Which is fine if scaling is not among your immediate goals.

Another approach is to rely exclusively on HTTP requests to emulate everything like authentication, form submission etc.. Very tedious to implement but it does scale like a charm.

 

[ThePacketeer]

The problem with Puppeteer, nightmare.js, and other libraries that rely on full browser emulation is that, if you screw up the implementation process, your bot will inevitably consume a fucktone amount of resources. Which is fine if scaling is not among your immediate goals.

Another approach is to rely exclusively on HTTP requests to emulate everything like authentication, form submission etc.. Very tedious to implement but it does scale like a charm.

We used to do it with curl indeed. However, with many websites going SPA with frameworks like react, next, vue, nuxt, etc, the html content most of the time only holds a div that is a placeholder for the app. Then you need to have something that can render framework. The easiest is a real browser.

And yes, puppeteer (chromium) can take some finetuning but we peaked at 4000 threads loadbalanced on 5 dedicated servers with 32gb ram and i7 cpu each, under ubuntu.Funnely enough the i7's outperform the xeons, probably because chromium uses some instruction sets that are better implemented on desktop cpus.

 

[karupoiss]

I mainly use puppeteer in nodejs (javascript) https://pptr.dev/
This combined with puppeteer-extra-stealth plugin to emulate unique users and avoid bot detection https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth
Then use ali's anyproxy to tunnel to a residential proxy and intercept ssl requests if needed

Wow, thanks for the link. for that extra stealth plugin. It seems like there is tons of information and a good place to start from (wiki). Seems very overwhelming atm but hopefully through time I can actually create that demo program there that automates making a screenshot of google Then.. social media sites, here I come!

 

[darkblackdark]

learn how to simulate specific web actions like login/forum post/etc. and observe the requests via a local proxy like burp. If ur lucky you can automate everything just with normal http requests if that site is not ajax based or has captchas. If its ajax based or too complicated to simulate the traffic then selenium webdriver with python bindings might be a solution.

 

[ragnarkar]

Hi!

I have a bit of a programming background - I have bachelors degree in informational technology but the bad thing is that I learned it 10 years ago and don't remember much of it (we learned java, python and php a bit).

I want to start making my own bots for stuff like social media automatization. Things like sending PM-s with multiple accounts at the same time, scraping content, posting, upvoting etc. I have read there are thousands of things you need to fake about browsers such as canvas and IP and user id etc.. I have no idea how or from where to start?

Any suggestions?

https://automatetheboringstuff.com/2e/chapter12/
That's where I'll start to learn the basics. After you master the basics, I'd just search stackoverflow for anything specific you want to do and hopefully someone has already done it before and you'll probably have the knowledge to re-engineer their code to fit your specific use case.

 

[Willyfish]

Custom bots:
Use: Selenium + python. Look for the book automate boring stuff with python, you'll learn a ton. You should learn python anyways if you plan on working on bots.
Use: Cheerio + nodejs (javascript). I find it more intuitive but you gotta learn javascript. You can also control the browser through Javascript.
Use: C# for more advanced bots, this allows you to have more control and to build interfaces as well.

Already available software:
Otherwise, you can simply rely on Zennoposter, it is good enough and helps you get started quicker. But there is a learning curve in that too.
The other advantage is that you can buy custom made templates for cheap. There also lots of free templates to learn from.
And more importantly, it is as intuitive as it gets. I usually refrain from using that kind of software but Zennoposter's UI design makes it easy to use and learn.
It is also highly customizable as you can include custom C# code snippets if you need granular control over the tasks that need to be accomplished.

P.S.: Sorry about the thousand edits

Did you try browser automation studio ? I was thinking of choosing it instead of zennoposter, I read it's even easier and as powerful ?

 

[HNB123]

BAS is very easy to use .
Just try it and you will adopt it

 

[daibu]

For a modern bot to have any chance of avoiding detection, it needs to be written using Puppeteer or Playwright and use evasion libraries. You'll need to handle browser fingerprinting with a service such as Multilogin and you'll need to simulate realistic human behavior. All of this is a huge endeavor, but doable if you're willing to put in the time. Learn in this order:

1) JavaScript
2) Node.js
3) Puppeteer

If you need to hide the fact that you're a bot, then you'll need to continue with:

4) Evasion libraries, such as puppeteer-extra-plugin-stealth
5) Human behavior libraries, such as ghost-cursor

If you need to appear as though you are many different users, then also learn about:

6) Browser fingerprinting

 

[StefoGalbu]

Personally I use selenium + C# and Visual Studio

 

[Daisychen509]

Social media automation bots are easy to build. But don't you think there are already so many social media bots out there. But, I you still want to build one, you can have a look at Botchief.

 

[Elvlin]

win automation is great, i use this to bot some games lol

 

[CoderFromHell]

I learned python selenium off of this 1, 8 hour YouTube tutorial made by this Indian guy.