Read a lot before you start, ie the docs of the packages you want to use, as well as the docs of the underlying frameworks etc. Also subscribe to as many security and anti-bot company blogs as you can to get insight from the other side. Like with "hacking" (rolleyes
) the best way to learn is to understand how to stop attacks first and then work backwards.
Everyone is going to do things a little differently, so I can only give my perspective.
My personal stack is:
Multilogin
Even though it's not perfect, it's the only solution I've found that comes close to doing what it says on the box. Plus it is the only anti-fingerprint solution that works with Puppeteer.
Puppeteer-Extra + Puppeteer-Extra-Stealth
Vanilla Puppeteer is very easy to detect, and if you want to use a plugin Captcha solver, you need Extra. Even with these, you'll need to chip away at certain things to harden it further.
https://github.com/berstend/puppeteer-extra
2Captcha + Puppeteer-Extra-Recaptcha
For solving Google Recaptchas
https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-recaptcha
4G Gateway + Script to rotate IP
Basically just any 4G hotspot that you can write a script to login + send a reboot command to - eg:
https://www.aliexpress.com/wholesale?SearchText=4g+hotspot
You need to be able to do this from a script, otherwise, you'll be doing it manually each time you want to rotate to a new identity.
An SMS solution + script to receive verification codes
No matter how good the bot is, you'll get phone verification requests eventually even if you've already verified at setup. Long term you'll want to get something like this
https://www.alibaba.com/product-detail/4g-lte-gsm-sms-gateway-gsm_62466674692.html but early on you can work with a cheap Android handset with a SIM expander and an IFTTT script to POST all SMS messages to your server. Then whenever you get a phone verification you just tell the bot to enter the phone number for that identity, then wait for the server to give it the code.
A backend API for identity management and content scraping
I built my backend on Laravel, included a screenshot below, but it becomes essential after you get to a certain number of accounts. To give a bit more context, by Puppeteer app requests a list of tasks from the API along with the account's login info etc and registers completion after they are done. It also accepts the inbound SMS and Email verification codes and passes these to the bot when it hits a verification wall. There's a lot of good plugins for Laravel that help with scraping etc as well.
For organisation, obviously you want to split all your logic out into "plugins" that you pull in as needed, otherwise, you'll end up with one huge script that's impossible to maintain.
With JS there isn't really a set structure, so it really comes down to preference. I typically mirror the organization of the frameworks I'm most familiar with...
However, for any node / ES6 app, you'll typically always want to start with an `app.js` entry point in your root folder, then abstract out your classes / mixins etc into logical folders.
As an example to get you started here's a screenshot of my current bot structure.
This is an example of my CLI entry point:
Hope that helps!
