Creating bots - the technical side

E

EpsilonMoves

Registered Member
Joined
Feb 16, 2021
Messages
60
Reaction score
9
Little about myself: B.Cs in computer science and have experience with android and web development (React, NodeJs, GCP) - but with zero experience with bots :(
I've watched some udemy courses about puppeteer but I was missing there the technical stuff and this is my question for this post:

I'm thinking of creating about 10 bots each one will be hosted on a private server so it will be using the local chrome browser, so (as a bot noob) is this solution cost effective or will it be expensive to do?
and if I get recommendations on the vps and ip services.
If there is any course or github repository with source code it will be a great help to me.
 
Advertise on BHW
Forget puppeteer.

If you want anything scalable, you have to program them using http requests.

Browser automation should only be used for things that are much harder with requests alone AND doesn't have to scale.

Since you already have experience with nodejs, it shouldn't be hard for you to pick up:

-Open your browser
-Ctrl + Shift + C
-Open the network tab
-Visit website and do the desired activity
-Replicate the requests using the http or http2 nodejs module based on whether the request was sent using http or http2.


For things which you can't access with a browser (games, apps etc), you should use a tool like wireshark to sniff the requests.
 
Bot for what? If for scraping, puppeteer may be overkill, http requests, websockets and dealing with API's should suffice for simple tasks.
Puppeteer is needed when you need content generated by JS in browser, that you can't retrieve with requests.
Chat bots? Minecraft bots?
 
Sorry mate I wasn't clear, bot for social sites (instagram and pinterest for start).
 
Yup, for those puppeteer is better. If you already have knowledge of JS, shouldn't be so hard to learn.
Mostly async functions, lots of awaits inside lol and some custom JS in browser to target DOM elements. Maybe sprinkle some Mutation Observer functions in there.
I did a custom chat bot using Puppeteer in the past, because I couldn't reverse engineer the websockets part of that particular site. It kinda eats resources(compared to simple requests), and sometimes failed to do stuff without warnings in my case (not because of resources), but it was the only way.
In Google Cloud, free tier, if I remember well, I've ran 7-8 instances of this without problems. I think it can run 10 as well.
 
selenium webdrivers might be also a good solution. You can automate firefox/chromium with it or a headless browser like phantomjs for better performance. You can use python interface
 
IcaroAugusto said:
Forget puppeteer.

If you want anything scalable, you have to program them using http requests.

Browser automation should only be used for things that are much harder with requests alone AND doesn't have to scale.

Since you already have experience with nodejs, it shouldn't be hard for you to pick up:

-Open your browser
-Ctrl + Shift + C
-Open the network tab
-Visit website and do the desired activity
-Replicate the requests using the http or http2 nodejs module based on whether the request was sent using http or http2.


For things which you can't access with a browser (games, apps etc), you should use a tool like wireshark to sniff the requests.
Click to expand...
very helpfull thanks
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top