Apologies in advance if this is not the right forum for this question.
I am looking to develop a script to scrape Instagram account details. The tool will take as input a list of account names and output basic information of that profile. I have a script that does this and it works as expected for a small input size but the problems start when the input list grows.
Ideally, I want the script to have at least an upper limit of 100k accounts. I am looking for tips on how I can bypass the technical limitations this problem poses. Any guidance would be highly appreciated!
I have tried switching user agents but the IP blocks happen at around 200 requests.
Some points I am interested in:
1. Would proxies work in this case?
2. Do I go for public proxies or private proxies?
3. How much time can I expect the script to take to scrape around 100k accounts?
4. How does the premium bulk scrapers avoid these bans?
I am looking to develop a script to scrape Instagram account details. The tool will take as input a list of account names and output basic information of that profile. I have a script that does this and it works as expected for a small input size but the problems start when the input list grows.
Ideally, I want the script to have at least an upper limit of 100k accounts. I am looking for tips on how I can bypass the technical limitations this problem poses. Any guidance would be highly appreciated!
I have tried switching user agents but the IP blocks happen at around 200 requests.
Some points I am interested in:
1. Would proxies work in this case?
2. Do I go for public proxies or private proxies?
3. How much time can I expect the script to take to scrape around 100k accounts?
4. How does the premium bulk scrapers avoid these bans?