Unsure how to deploy scraping service

DogNamedRottles said:

I'm building a scraping bot for a friend of mine to monitor his amazon products. His inventory is quite small (30-50 products) and the amount of times he has to check his products is infrequent (once a month or quarter)

To provide more context, I have the service built out and it's able to navigate to Amazon and the product pages without a captcha block so long as I am using my home ip and the browser stealth plugin is enabled for Playwright. The issue I'm concern about is when this is deployed to a hosting platform like DigitalOcean or Railway that Amazon will start blocking requests more often because the ip will now be from a datacenter. I thought of serveral solutions around this.

1.) If my buddy is not planning on monetizing this, then have him take advantage of his office ip and deploy the service on a physical device. His inventory isn't that large and the monitoring would be so infrequent that Amazon wouldn't detect any bot activity. I would have to install a physical server though and setup networking so that he can access the service himself.

2.) Deploy to a VPS and use a rotating residential proxy. That I would think would be enough to handle the anti-bot detection. The drawbacks are that proxies are expensive and I would have to implement some retry capability to allow for resilience. And I don't think friend will be able to justify the cost on a monthly basis. However, I would suggest to him if he's planning on monetizing this with his partners since then we can justify the cost since now we have people paying for it.

However, my question is if I'm going in the right direction with this? Or am I overthinking the potential issues?

Click to expand...

Ashk881 said:

No the issues you said are legit and everybody who has to do infrequent scraping has to deal with this problem.

The way I like to go with this is to have a consumer computer running 24/7 that acts as a homeserver. No need to get expensive enterprise grade servers to run simple software. Just get a used laptop or tower PC and install a Linux server distro like Ubuntu Server or fedora on it.

You can even use ancient devices for this. Most of this kind of software is lightweight that you can use processors from 10 years ago and sufficient DDR3 RAM.

Set your app as a docker container, keep the device connected to the internet and you've a cheap server up and running.

You can even go fancy and install Tailscale on the homeserver and your friend's laptop so they can access and control it remotely. No static IP needed.

Click to expand...

DogNamedRottles said:

Ah I see. I'm glad I'm not the only one who's going through this.

You mentioned about Docker. Do you have a suggestion for an affordable way to store images? I plan on using Github Actions as my CI/CD pipeline and I want to at least be able to switch out image versions just in case one of the deployments goes to shit.

Click to expand...