Scrapping data from a website for 7 days

i have a script that scrapes data from a website, and I’d like to run it continuously for about 7 days.
What would you recommend in terms of setup to ensure it runs reliably, and what best practices should I follow to reduce the risk of being blocked?
i also want to run the same script in many servers, so i can speed up the process.

Depends on how you're scraping - is it through a server side script or something you're running on a machine?

If it's a server side, just use cron to run it automatically and update the script to have logging so that if it fails, the script can be executed again/repeated until it succesfully completes each day.

If you're running software, I'd run it on a vps and use the OS' scheduler like windows tasks to initiate it.

If you want to scale across many servers, split the workload cleanly so each worker handles a separate batch without duplication.
Use centralized storage, heartbeat monitoring, and retry logic with backoff for reliability.
For blocking risk, focus on compliance, sane concurrency, and identifying yourself properly where appropriate.

Get a vps or some virtual machine somewhere, run it there?

khalidou said:

i have a script that scrapes data from a website, and I’d like to run it continuously for about 7 days.
What would you recommend in terms of setup to ensure it runs reliably, and what best practices should I follow to reduce the risk of being blocked?
i also want to run the same script in many servers, so i can speed up the process.

Click to expand...

To reliably run your scraping script for 7 days across multiple servers, use cloud servers with Docker and a robust database. Implement a task queue like Celery to distribute the workload. To prevent blocking, rotate residential proxies such as Bright Data or Smartproxy, use random user agents, and set random delays between requests. Limit the request rate, manage cookies, and consider using captcha solving services. Monitor for errors and respect the robots.txt file. Distribute tasks across servers, manage shared proxies, and synchronize data to a centralized database.

khalidou said:

i have a script that scrapes data from a website, and I’d like to run it continuously for about 7 days.
What would you recommend in terms of setup to ensure it runs reliably, and what best practices should I follow to reduce the risk of being blocked?
i also want to run the same script in many servers, so i can speed up the process.

Click to expand...

You need a VPS and you might need proxies (Residential probably).

Contact me if you need further assistance.

khalidou said:

i have a script that scrapes data from a website, and I’d like to run it continuously for about 7 days.
What would you recommend in terms of setup to ensure it runs reliably, and what best practices should I follow to reduce the risk of being blocked?
i also want to run the same script in many servers, so i can speed up the process.

Click to expand...

this should be more of a small task which runs on schedule rather than something that just runs blindly like a script. there needs to be some error checking, logging, timeouts nd auto reccovery so that failure of one does not bring down the entire 7 day task. in case of multiple servers, there needs to be separate tasks assigned to each of them.