Best proxy solution for scraping 1M Instagram profiles daily without login

Hi!
I need to scrape 1M Instagram profiles each day without logged in accounts through Instagram only allows an IP to access this endpoint a few times per time interval before it gets blocked.
Because of this, many IPs are needed in the proxy network.

What is the best and cheapest solution for this?
I know there are providers like BrightData that cost a fortune to do it, are there cheap but working alternatives out there?
I tested some providers from the marketplace here but either they are blocked already or get blocked after some time because the IP pool is so small.

I'd be very grateful for some working tips.

Are you sure it's the IP getting blocked? It could also just be your user agent given that most 4g providers have millions of IPs in their pool. I find it hard to believe that you've gone through all of them.

Alternatively, this method sounds like it would be a massive pain in the ass. Why not just scrape using an actual account and literally exponentially increase your results?

I'd recommend you to try out 4G mobile proxies as mobile IPs are way more dynamic than standard IPs; are rotative in nature adding to their anonymity/credibility and making'em the hardest to be detected, flagged or banned. What you can do is go for a trial and see for yourself before investing in a product.

AllOutAnime said:

Hi!
I need to scrape 1M Instagram profiles each day without logged in accounts through

Code:

https://www.instagram.com/instagram/?__a=1

Instagram only allows an IP to access this endpoint a few times per time interval before it gets blocked.
Because of this, many IPs are needed in the proxy network.

What is the best and cheapest solution for this?
I know there are providers like BrightData that cost a fortune to do it, are there cheap but working alternatives out there?
I tested some providers from the marketplace here but either they are blocked already or get blocked after some time because the IP pool is so small.

I'd be very grateful for some working tips.

Click to expand...

a 4G proxy with a rotation api will do, once you get blocked you can rotate it and get a new clean ip. there's proxyguyz and litport also anonymous-proxy you'll get virgin ips every now and then.
i'm not affiliated with any of the mentioned providers above so try them at your own risk.

Thank you very much for all the replies!

GreyHatNik said:

Are you sure it's the IP getting blocked? It could also just be your user agent given that most 4g providers have millions of IPs in their pool. I find it hard to believe that you've gone through all of them.

Alternatively, this method sounds like it would be a massive pain in the ass. Why not just scrape using an actual account and literally exponentially increase your results?

Click to expand...

Yes, I tried different useragent and header combinations and once an IP receives the login page, it will always receive the login page for some time.
I don't burn through all their IPs immediately. Many providers have a success rate of around 70% at the start but as time goes on this drops to 20% because I think they don't have enough IPs to sustain that load and Instagram blocks the IPs.

Using accounts to scrape is also hard because accounts get banned as well and I would constantly need to re-supply accounts to keep it up.
I assume for my scraping I would need at least 5k of them active at all times.
Instagram accounts are not cheap as well and I would need residential IPs to run them.

Sexat said:

You can try rotating proxy or socks if static ips/proxies not working.

Click to expand...

I agree but which provider is even good for this?

MoonStarz said:

I'd recommend you to try out 4G mobile proxies as mobile IPs are way more dynamic than standard IPs; are rotative in nature adding to their anonymity/credibility and making'em the hardest to be detected, flagged or banned. What you can do is go for a trial and see for yourself before investing in a product.

Click to expand...

MasonF said:

a 4G proxy with a rotation api will do, once you get blocked you can rotate it and get a new clean ip. there's proxyguyz and litport also anonymous-proxy you'll get virgin ips every now and then.
i'm not affiliated with any of the mentioned providers above so try them at your own risk.

Click to expand...

Mobilie proxies could work but in order to reah 1 million requests per day, I would need to make 12 requests per second which means I would need at least 12 of those mobile proxies. Then switching IPs takes 10 seconds to 1 minute so I would need to use even more requests per second and it seems buying 10 - 20 mobile proxies for this is equal to using BrightData.

AllOutAnime said:

Thank you very much for all the replies!


Yes, I tried different useragent and header combinations and once an IP receives the login page, it will always receive the login page for some time.
I don't burn through all their IPs immediately. Many providers have a success rate of around 70% at the start but as time goes on this drops to 20% because I think they don't have enough IPs to sustain that load and Instagram blocks the IPs.

Using accounts to scrape is also hard because accounts get banned as well and I would constantly need to re-supply accounts to keep it up.
I assume for my scraping I would need at least 5k of them active at all times.
Instagram accounts are not cheap as well and I would need residential IPs to run them.

I agree but which provider is even good for this?


Mobilie proxies could work but in order to reah 1 million requests per day, I would need to make 12 requests per second which means I would need at least 12 of those mobile proxies. Then switching IPs takes 10 seconds to 1 minute so I would need to use even more requests per second and it seems buying 10 - 20 mobile proxies for this is equal to using BrightData.

Click to expand...

Dude, one account should easily be able to scrape like 20k users in a day. Unless you're employing a shit ton of filters and have several api calls to scrape just one profile, you definitely won't need 5000 accs to scrape 1M users.