Large scale puppeteer bots

[rampzeus]

I am planning to run a couple of hundred bots with puppeteer but I am worried about the ram/cpu bottleneck.

It costs around 500-800 ram for each instance, this goes up quite fast when scaling hundreds of bots.

These problems would not be a problem with using an api but I am unable to find a reverse engineered facebook api.

Anyone else doing something similar and has some tips/recommendations?

 

[monst3rwint3r]

Of course, you will run into a limit on RAM. Requires horizontal scaling with multiple servers.

 

[seancoder]

Came across something similar sometime ago. How can one get more info regarding horizontal scaling ?

 

[AlexAkimbo]

Have you tried k8s/docker? You will need some sort of load balancing that can be taken care of with nginx.

 

[seancoder]

Have you tried k8s/docker? You will need some sort of load balancing that can be taken care of with nginx.

Interesting regarding the load balancing aspect will aslo have a look at that.

 

[AlexAkimbo]

Interesting regarding the load balancing aspect will aslo have a look at that.

Basically you'll need proper chrome docker image and nginx as load balancer in front. Puppeteer will send requests to nginx which will forward them to free container. You can use least_con or pre_req_health_check to drive load balancer.

Also, if you are using python you can create VPS cluster, install Ansible, setup Tower (or any other solution) and then use job scheduling.

 

[rampzeus]

Have you tried k8s/docker? You will need some sort of load balancing that can be taken care of with nginx.

I did but quite hard to mask your fingerprint and the fact that you are headless/xvfb. Ban rate was through the roof.

So far best I have found is a workerpool with headfull browser. This gives decent flexibility but doesn't have the fingerprint problem.

 

[AlexAkimbo]

I did but quite hard to mask your fingerprint and the fact that you are headless/xvfb. Ban rate was through the roof.

So far best I have found is a workerpool with headfull browser. This gives decent flexibility but doesn't have the fingerprint problem.

I wasn't talking about base docker images in my post. You can find chrome images (or you can make them) with CDP enabled. That way you can run headful Chrome in docker container and drive it directly from puppeteer via CDP or any other approach. Check out KASM images on dockerhub

 

[rigourousplans109]

I wasn't talking about base docker images in my post. You can find chrome images (or you can make them) with CDP enabled. That way you can run headful Chrome in docker container and drive it directly from puppeteer via CDP or any other approach. Check out KASM images on dockerhub

You mean these?

https://hub.docker.com/r/kasmweb/firefox
https://hub.docker.com/r/kasmweb/chrome

 

[AlexAkimbo]

You mean these?

https://hub.docker.com/r/kasmweb/firefox
https://hub.docker.com/r/kasmweb/chrome

Yes. These were good when I tested them but haven’t put them in production. Ultimately I ended up making my own image and putting it up on k8s

 

[rigourousplans109]

I wasn't talking about base docker images in my post. You can find chrome images (or you can make them) with CDP enabled. That way you can run headful Chrome in docker container and drive it directly from puppeteer via CDP or any other approach. Check out KASM images on dockerhub

I already have small number of twitter accounts running using different methods while avoiding the use of paid anti detect browser. I'm at a crossroads because I cannot realistically scale up and before I buy and software I want to try headful chrome/firefox instances in docker. You think this is worth exploring for twitter accounts? Someone on this forum said they had 1k accounts on docker and that's why I'm giving it a shot before moving to anti detect browsers.

I want to work with 300-400 bots. I think I can make do with 250 if it becomes too much.

 

[rigourousplans109]

Ok, Kasm is very good. I'm will add 25 accounts and see for a month if it works.

 

[vittorianicolosi1994]

Almost looking tò achive the Same and i i have only 32 GB.
How did you resolve ?

 

[blazedat]

Ok, Kasm is very good. I'm will add 25 accounts and see for a month if it works.

Curious to see how the test ended, did the KASM image perform good?

 

[vittorianicolosi1994]

Do you have the right hardware ; creation of account can be done on iPhone 6s with cloned app.


Once you done creation Is more Easy to avoid shadowban

1 PC Is enough as server to manage your smartphones.
Use stealth browser.


Each proxy providers are different, someone allow ton run 15 acc ish for proxy


Let's talk about limit of iPhone:

:

"One Device + 3 Cloned apps on that Device + 1 instagram accounts per clone = 3Accounts

per 4g ROTATING Proxy.



Each 10 days add 1 IG account per CLONE.



Your finale result Will be

Three Devices+ 3 Cloned Apps per Device + 2 Instagram accounts per clone = 18 Accounts per 4g ROTATING Proxy.

Once you done It move this 18 Accounts tò your server PC, get Gologin and buy a bot for web browser.

I Have a question, what if i dont own expensive server PC and i want manage 200 account ?

I Guess i Need create my own Chrome headless docker image with CDP enabled. Where do i find some info ?