resources to learn legit bot making

[rndom666]

hey guys, i'm really interested in learning to best approach scraping websites. i know there are some veterans out there who might have learned a thing or two, and it's always good to stand on the shoulders of giants.

so, i'm hoping to get some insights into figuring out website apis quickly - tips, tricks, maybe some interesting anecdotes.

edit: i primarily use python and linux.

cheers

 

[SalvadorDale]

Try channel Bot empire on YT, or search for Zennopoter guides

 

[rndom666]

Try channel Bot empire on YT, or search for Zennopoter guides

thanks man, is the zennopoter manual still relevant? genuinely curious, because i see it's copyrighted 2011. or maybe i have the wrong one.

i did look at stuff on youtube, but seemed kinda basic. i'm trying to figure out best practices to save time, maybe idea like bots that make bots i have no clue. pretty much open to anything actually.

 

[SalvadorDale]

thanks man, is the zennopoter manual still relevant? genuinely curious, because i see it's copyrighted 2011. or maybe i have the wrong one.

i did look at stuff on youtube, but seemed kinda basic. i'm trying to figure out best practices to save time, maybe idea like bots that make bots i have no clue. pretty much open to anything actually.

Python it's the easiest way to build complex bots with multiple functions

 

[uncutu]

i really like zennoposter. i bought it many years ago a because i didnt know how to make browser automation bots. and thats what ZP positions itself as. no-code drag & drop browser automation (you can still add raw code if you know it)

i still use it today. it's still great. i showed it to some friends and they had a hard time with the learning curve. maybe its just because i am a senior with it now, but i found it very straight forward. sure some very complex stuff was tricky, but via trial and error and old youtube videos you can do it. it's mostly drag & drop interface with some variables. yes the YT videos are old, but it's all the info is still current to running the software. they also have a members forum thats usually really good at helping users perform complex tasks and even sharing template files.

the other option is learning python selenium or requests. many youtube videos here. for example search youtube for automate tinder with selenium. follow along step by step. this was my first intro to selenium and i did really good with it. having years of ZP botting under my belt helped.

so why may you chose Python over ZP. python is free. lighter on the resources. but higher learning curve. but python has more applications outside of botting whereas zp doesnt. i can make a super complex script in ZP in a day whereas it would take me weeks of googling spaghetti code on python. i like ZP cus it's drag & drop, no-code.

the main reason i even considered python was i got my hands on a bot a competitor made for a site i was botting. it was made with python requests. 1000x more efficient than what ZP was doing for me. blew my mind.

 

[rndom666]

i really like zennoposter. i bought it many years ago a because i didnt know how to make browser automation bots. and thats what ZP positions itself as. no-code drag & drop browser automation (you can still add raw code if you know it)

i still use it today. it's still great. i showed it to some friends and they had a hard time with the learning curve. maybe its just because i am a senior with it now, but i found it very straight forward. sure some very complex stuff was tricky, but via trial and error and old youtube videos you can do it. it's mostly drag & drop interface with some variables. yes the YT videos are old, but it's all the info is still current to running the software. they also have a members forum thats usually really good at helping users perform complex tasks and even sharing template files.

the other option is learning python selenium or requests. many youtube videos here. for example search youtube for automate tinder with selenium. follow along step by step. this was my first intro to selenium and i did really good with it. having years of ZP botting under my belt helped.

so why may you chose Python over ZP. python is free. lighter on the resources. but higher learning curve. but python has more applications outside of botting whereas zp doesnt. i can make a super complex script in ZP in a day whereas it would take me weeks of googling spaghetti code on python. i like ZP cus it's drag & drop, no-code.

the main reason i even considered python was i got my hands on a bot a competitor made for a site i was botting. it was made with python requests. 1000x more efficient than what ZP was doing for me. blew my mind.

appreciate the info on zennoposter. yeah, im mostly interested in using python since i already know how. i just never used it for scraping and bots - i mostly use it for data engineering stuff.

 

[rndom666]

Python it's the easiest way to build complex bots with multiple functions

do you have any examples of this? basically im trying to figure out how to best approach things like websites that use random characters in their url path, how to best inspect their html what kind of things might be of interest and not, etc.

 

[uncutu]

do you have any examples of this? basically im trying to figure out how to best approach things like websites that use random characters in their url path, how to best inspect their html what kind of things might be of interest and not, etc.

youtube watch any python video for tinder (or any site)automation they cover all of that step by step

 

[rndom666]

youtube watch any python video for tinder (or any site)automation they cover all of that step by step

well for example, in order to save time it would be good to know what circumstances it's better to use reqeusts/httpx vs scrapy vs selenium. when and why, downfalls - maybe some examples of things you've done with each.

 

[rndom666]

also interested in thoughts on headers and what to watch out for maybe

 

[Lipstick Spoiler]

Wha

i really like zennoposter. i bought it many years ago a because i didnt know how to make browser automation bots. and thats what ZP positions itself as. no-code drag & drop browser automation (you can still add raw code if you know it)

i still use it today. it's still great. i showed it to some friends and they had a hard time with the learning curve. maybe its just because i am a senior with it now, but i found it very straight forward. sure some very complex stuff was tricky, but via trial and error and old youtube videos you can do it. it's mostly drag & drop interface with some variables. yes the YT videos are old, but it's all the info is still current to running the software. they also have a members forum thats usually really good at helping users perform complex tasks and even sharing template files.

the other option is learning python selenium or requests. many youtube videos here. for example search youtube for automate tinder with selenium. follow along step by step. this was my first intro to selenium and i did really good with it. having years of ZP botting under my belt helped.

so why may you chose Python over ZP. python is free. lighter on the resources. but higher learning curve. but python has more applications outside of botting whereas zp doesnt. i can make a super complex script in ZP in a day whereas it would take me weeks of googling spaghetti code on python. i like ZP cus it's drag & drop, no-code.

the main reason i even considered python was i got my hands on a bot a competitor made for a site i was botting. it was made with python requests. 1000x more efficient than what ZP was doing for me. blew my mind.

What are your views on browser automation tool?

 

[Smackhead]

My preference is usually puppeteer or selenium but it really depends on the sites. I have never used pre build scraping tools so probably bias.

There might be times that an out of the box tool will be faster but eventually you are going to run into something it can't do.

If you need to write something highly optimised with little overhead then http request are going to he your best bet but this can often be complex, alot of sites will protect against this so you'll need an understanding of headers (start by looking at your network tab in the browser. You can extract any request from there along with its headers. Start to remove headers to figure out what is required and then work out how the UI is generating those headers). There will be times where this is very easy.

If a site is rendered on the server side then you can simply parse the html, i think this is where out of the box scrapers are useful.. but most sites these days will generate data on the fly in which case this isn't possible.

9 times out of 10 you'll be able to do what you need with puppeteer / selenium and will also develop the skills required to pivot should you need something different. As for langauge my personal preference is c# but there are also javscript and python versions available.

Chrome has an awesome dev tool for recoding your actions and exporting into puppeteer which is a gear starting point and saves you having to endlessly inspect the DOM.

 

[Surfinite]

If you have knowledge in programming, it is better to use Selenium. An important point in automation is to implement human behavior (mouse trajectory, text input, etc.)

 

[vuminh075]

You can find it on youtube or stackoverflow have many sample in there

 

[rndom666]

My preference is usually puppeteer or selenium but it really depends on the sites. I have never used pre build scraping tools so probably bias.

There might be times that an out of the box tool will be faster but eventually you are going to run into something it can't do.

If you need to write something highly optimised with little overhead then http request are going to he your best bet but this can often be complex, alot of sites will protect against this so you'll need an understanding of headers (start by looking at your network tab in the browser. You can extract any request from there along with its headers. Start to remove headers to figure out what is required and then work out how the UI is generating those headers). There will be times where this is very easy.

If a site is rendered on the server side then you can simply parse the html, i think this is where out of the box scrapers are useful.. but most sites these days will generate data on the fly in which case this isn't possible.

9 times out of 10 you'll be able to do what you need with puppeteer / selenium and will also develop the skills required to pivot should you need something different. As for langauge my personal preference is c# but there are also javscript and python versions available.

Chrome has an awesome dev tool for recoding your actions and exporting into puppeteer which is a gear starting point and saves you having to endlessly inspect the DOM.

this is super helpful man thanks, i will try out the header technique to learn a bit more.