Heavy Scraping on Insta - Proxy Suggestion

Hi all,

I'm looking to do heavy scraping with Instagram on a daily basis. I already have residential proxies set up which are rotating every few mins. On average, the IP gets blocked after around 2 mins so most of the time my scraper is at idle. I want to have multiple proxies running simultaneous.
I checked our crawlera on the C10 plan which is pretty good, but I hit 150K requests in just a few days. They get pretty expensive when you start getting in to the millions of requests.

Does anyone have any suggestions to keep costs down? I believe a lot of guys use free proxies and just continuously rotate them. Any sources for good free proxy lists? Could be socks or http, doesn't matter.

Thanks

What do you scrape on instagram? Expired domains?

Going to be really hard, if not nearly impossible to scrape millions of pages on IG using free proxies. Think of free proxies as very slow, unreliable and heavily abused. If your other proxies are only lasting 2 minutes, have you tried to slow your bot down to see if you can scrape more pages before being blocked? Sometimes a bit slower gets you more than just trying to go balls out crazy fast.

Id love to know too.

what tool / bot are you using

proxyguys said:

Going to be really hard, if not nearly impossible to scrape millions of pages on IG using free proxies. Think of free proxies as very slow, unreliable and heavily abused. If your other proxies are only lasting 2 minutes, have you tried to slow your bot down to see if you can scrape more pages before being blocked? Sometimes a bit slower gets you more than just trying to go balls out crazy fast.

Click to expand...

I agree, it would have to be designed in a way to delete any poor performing proxies and only use the ones which are active. I'm actually going to try and slow it down a bit and see, but I think they're blocking based on the quantity and number of request coming from a single IP, but I'll do a test to see. Do you have any good scraping proxies which are cheaper than crawlera?

proxygo said:

what tool / bot are you using

Click to expand...

It's a bot I had developed for personal use.

indeed97 said:

It's a bot I had developed for personal use.

Click to expand...

Why not you "tame" it down a bit and slow your scraping speed. You might scrape fewer data/pages, but in the long run you might end up scraping more.

Chris.Roark said:

Why not you "tame" it down a bit and slow your scraping speed. You might scrape fewer data/pages, but in the long run you might end up scraping more.

Click to expand...

Yea, I can do that, but I'm looking to perform 1-2M requests a day. Even if I tame it down to avoid the bans, I need additional scripts running in parallel to get me greater return in a shorter time span. So I'm thinking of some alternative methods which involve alternative proxies.