CoderFromHell
Power Member
- Mar 19, 2019
- 727
- 341
How well would these perform against sneaker sites?
How should I get started if I want to learn browser automation?
- Thread starter karupoiss
- Start date
How well would these perform against sneaker sites?
daibu
Junior Member
- Aug 7, 2020
- 101
- 64
It would depend on the particular site, but most bot detection is handled through third-party services, such as DataDome. Bypassing the detection of such services is a cat-and-mouse game that could require a tailored solution on top of the stack I mentioned, depending on the state of things.
Magic555
Regular Member
- Apr 28, 2017
- 350
- 208
For basics you can start using selenium and automate sites that don't have fingerprinting and move your way up. You will come to a point where you'll realize its very hard to automate platforms with good protection like facebook and google services. This is when you need to up your spoofing game.
Things to keep in mind,
Secure platforms use complex detection systems there are things like audio context and hardware concurrency for example.
Things to keep in mind,
Secure platforms use complex detection systems there are things like audio context and hardware concurrency for example.
SpivBlack
Regular Member
- Apr 23, 2020
- 217
- 70
When building browser-based bots try using docker to avoid bot detection. Spin up each container for each account with its own volume. Using headless chrome, not chromium the fingerprinting will not be a problem. All data will be stored in a volume and will look like a real person.
daibu
Junior Member
- Aug 7, 2020
- 101
- 64
I think this is misleading. It might be a fine solution on sites that aren't trying very hard, but there are some very simple checks to detect if a user is running in headless mode (and only bots use headless). Please consult this Stack Overflow question:
https://stackoverflow.com/questions/55364643/headless-browser-detection
There may be a foolproof way to make headless Chrome appear as though it is headless, but it would be very complex and isn't necessary. Instead, you can run a virtual display using software such as Xvfb inside the docker container and then run Chrome normally.
SpivBlack
Regular Member
- Apr 23, 2020
- 217
- 70
Have you tried or are you going off what someone has said? With the dev tools alone you can control chrome as if you're a person. I have used this method for many projects. On high profile sites, ie: Supreme Adidas Nike, and Newegg. If the person would ask for the container, then I would have provided one. But the path to building software should no be handed to someone they need to learn and ask questions.
Best Regards
daibu
Junior Member
- Aug 7, 2020
- 101
- 64
It's based on experience and trusting what is documented in many places. Most of the points made by the top answer in that article are verifiable by running simple commands in the console. If you can examine those variables, then a website can too.
These web automation frameworks are created for the purpose of testing web applications. They don't make any special attempt to hide the fact that they're being used. Steps definitely need to be taken before a bot can reasonably appear to be human.
Here are a couple of old but famous articles in the space debating whether it's even possible to fully disguise the fact that you're running in headless mode:
https://antoinevastel.com/bot detection/2018/01/17/detect-chrome-headless-v2.htmlhttps://intoli.com/blog/not-possible-to-block-chrome-headless/
Long story short, you can skip the headache by simply running a virtual display in your Docker container. It's an easier (and more robust) way to address things.
SpivBlack
Regular Member
- Apr 23, 2020
- 217
- 70
What frameworks are you talking about?
Chrome dev tools protocol is what chrome uses from the GUI. You can take control of the V8 engine and send back a response that you want. You can take control of the network stack and modify the payload.
Code:
https://chromedevtools.github.io/devtools-protocol/Remember all website server is nothing more than GET, POST, PUT, DELETE. And the javascript for detecting if you are a bot runs on your device, not theirs. Send back the responses that they are looking for and you will be fine.
As for the display I have provided a Dockerfile that will compile with X11 and a init system that will allow for running more than one process in the container. If some want to know how to add Chrome to a container as well. They must ask.
Dockerfile:
Code:
FROM debian:stretch-slim
ENV DINIT=1.2.4
ADD https://github.com/Yelp/dumb-init/releases/download/v${DINIT}/dumb-init_${DINIT}_amd64.deb /tmp/dumb-init.deb
RUN apt-get update && apt-get install -y x11-apps &&\
dpkg -i /tmp/dumb-init.deb &&\
apt-get autoclean && apt-get autoremove &&\
rm -rf /tmp/* /var/lib/apt/lists/* /var/tmp/* /root/* /usr/share/doc/* /usr/share/info/*
RUN useradd -ms /bin/bash user
ENV DISPLAY :0
USER user
ENTRYPOINT ["/usr/bin/dumb-init", "--"]
CMD ["/bin/bash"]Best Regards
TheOverlord
Regular Member
- Sep 1, 2014
- 302
- 194
to add to all of this it would help to learn a bit of web programming if you don't already know it. Just the basics, like how HTML pages are structured and recognizing how JS and CSS interact with the site. Honestly just diving into a project automating something may be a good way to learn all of the basics, and take side diversions when needed
daibu
Junior Member
- Aug 7, 2020
- 101
- 64
Web automation frameworks. Selenium, Puppeteer, Playwright, etc.
I think you mean requests, right? In any case, this requires knowing what the proper request for each call is supposed to look like. What if the following code is used to fetch a list of items that you're interested in?
Code:
const fetchOptions = {
body: JSON.stringify({ isAutomated: navigator.webdriver }),
};
const cars = await fetch('https://example.com/get-items-if-user-is-not-automated', fetchOptions);
// Add cars to listSure, you can examine the client-side code, figure out what they're doing, intercept the request and change the body to "{ isAutomated: undefined }", but why resort to all that when you can simply add a virtual display to your Dockerfile and run headful? Not to mention, it addresses other leaks and won't break if they change their code.
In your Dockerfile above, just add xvfb to your apt-get install, and throw this in your ENTRYPOINT file:
Code:
Xvfb :99 -screen 0 1024x768x24It should be as simple as that.
It's possible I'm not understanding what you mean or vice-versa. I apologize if it's me, but I don't see how anything is simpler than what I'm suggesting.
Last edited:
SpivBlack
Regular Member
- Apr 23, 2020
- 217
- 70
Man, I love our conversation, no apology need. We have different views and we talked about them this is how people learn.
I don't use a framework for automation. I think that's where the disconnect is coming from.
In any event, the reason why I don't use a display driver, is the servers I use don't have a video card there just compute nodes. CPU, RAM, and a small SSD the setup and configuration are done by Ansible. So when I write the code, using Rust and Go. I assume no video card even if there is one. To allow for clients to use any service they want.
This will allow deployment to any Kubernetes or Docker Swarm without any support call
although I like Docker Swarm some people prefer Kubernetes.Finally with are different points of view people reading this post will have two valid paths that they can pick from.
Best Regards
sfnet
Junior Member
- Feb 17, 2020
- 119
- 38
@daibu and @SpivBlack I've found your conversation really interesting.
So basically @SpivBlack you're not using Puppeteer or any of the automation libraries. You've built your own bot with Go and Rust and bundled it with chrome in docker. This feels more robust but definitely difficult to implement.
And @daibu, you're using Puppeteer + headless chromium with xvfb in docker to simulate a headful browser to avoid detection. This seems like an easier solution.
My question is, why do you use the containers (docker)?
I've run some bots where I used PM2 to basically execute multiple instances and automatically restart the bots in case of an error.
Do I need to use docker + pm2 to be more robust? (if I were to use your solution @daibu )
So basically @SpivBlack you're not using Puppeteer or any of the automation libraries. You've built your own bot with Go and Rust and bundled it with chrome in docker. This feels more robust but definitely difficult to implement.
And @daibu, you're using Puppeteer + headless chromium with xvfb in docker to simulate a headful browser to avoid detection. This seems like an easier solution.
My question is, why do you use the containers (docker)?
I've run some bots where I used PM2 to basically execute multiple instances and automatically restart the bots in case of an error.
Do I need to use docker + pm2 to be more robust? (if I were to use your solution @daibu )
SpivBlack
Regular Member
- Apr 23, 2020
- 217
- 70
The main reasons I use containers are:
1) Deployment:
- Easy to package up for distribution
- Can run on any Kubernetes Cluster, Docker Swarm or even Docker Compose.
- This allows scale with a good set of proxies as well as a few cloud providers. The sky is the limit.
- You can attach a volume that will persist Storing all the cookies and other data need. This helps in warming up your bot ie: logging into the site you want with a profile etc.
- You can try many configurations at once and find out which will work best. Also distroying what doesn't work and keeping what does.
- For as little as $0.0075 an hour you can safely run 3-5 containers.
- No need to leave your computer(s) running.
Best Regards
sfnet
Junior Member
- Feb 17, 2020
- 119
- 38
Thanks for the explanation.
Which cloud service are you using to host the containers? Is it AWS EC2? Because a similar pricing of what I've seen on AWS only gives about 1 GB of Ram.
I suppose you're not running puppeteer nor chromium so that's enough for your containers?
Question for @daibu.. you on the other hand are running a full blown puppeteer + chromium + xvfb on your containers right?
How much do you spend per hour on your servers and what specs are those?
Also, how many containers are you able to run per server?
And, do you also store cookies and data in your containers? I've found that puppeteer starts with a fresh slate (as if I'm running in incognito) and never persists my cookies. I'm wondering how you are able to achieve that if you do this.
Sorry i have so many questions.. I'm trying to optimize my bots.
Thanks
SpivBlack
Regular Member
- Apr 23, 2020
- 217
- 70
For my own projects, I use:
- DigitalOcean
- Vultr
- Linode
I use the smallest one they have. At least a 1GB ram 1VCPU
Depends on the project but for Sneaker Bots 3 per VPS
Yes, in the volume attached to the container.
I don't know how Puppeteer works. Since I am using ChromeDev Tools protocol. I think @daibu would be better at answering that part.
Ask me all the questions you have if I can answer them I will.
D
Deleted member 977120
Guest
RPA or Selenium
daibu
Junior Member
- Aug 7, 2020
- 101
- 64
Oh, I see. Yeah, I've never used the CDP directly. Maybe that eliminates some of the headless issues.
Not headless. I use { headless: false } in my launch options. That's the reason for running a virtual display. Also, you'll want to use Chrome instead of Chromium if possible.
Use the userDataDir launch option to store cookies.