1. This website uses cookies to improve service and provide a tailored user experience. By using this site, you agree to this use. See our Cookie Policy.
    Dismiss Notice

Heavy Scraping on Insta - Proxy Suggestion

Discussion in 'Proxies' started by indeed97, Nov 8, 2019.

  1. indeed97

    indeed97 Registered Member

    Joined:
    May 6, 2009
    Messages:
    71
    Likes Received:
    7
    Hi all,

    I'm looking to do heavy scraping with Instagram on a daily basis. I already have residential proxies set up which are rotating every few mins. On average, the IP gets blocked after around 2 mins so most of the time my scraper is at idle. I want to have multiple proxies running simultaneous.
    I checked our crawlera on the C10 plan which is pretty good, but I hit 150K requests in just a few days. They get pretty expensive when you start getting in to the millions of requests.

    Does anyone have any suggestions to keep costs down? I believe a lot of guys use free proxies and just continuously rotate them. Any sources for good free proxy lists? Could be socks or http, doesn't matter.

    Thanks
     
  2. dekadent30

    dekadent30 Senior Member

    Joined:
    Nov 10, 2008
    Messages:
    811
    Likes Received:
    258
    What do you scrape on instagram? Expired domains?
     
  3. proxyguys

    proxyguys Junior Member Marketplace seller

    Joined:
    Jul 16, 2019
    Messages:
    121
    Likes Received:
    78
    Occupation:
    Proxy Scientist
    Location:
    California, USA
    Home Page:
    Going to be really hard, if not nearly impossible to scrape millions of pages on IG using free proxies. Think of free proxies as very slow, unreliable and heavily abused. If your other proxies are only lasting 2 minutes, have you tried to slow your bot down to see if you can scrape more pages before being blocked? Sometimes a bit slower gets you more than just trying to go balls out crazy fast.
     
  4. Seagate44

    Seagate44 Power Member

    Joined:
    Aug 14, 2016
    Messages:
    540
    Likes Received:
    102
    Id love to know too.
     
  5. proxygo

    proxygo Jr. VIP Jr. VIP

    Joined:
    Nov 2, 2008
    Messages:
    33,734
    Likes Received:
    12,896
    Gender:
    Male
    Occupation:
    Proxies Back Connect
    Location:
    UK - ALWAYS ON BHW
    Home Page:
  6. indeed97

    indeed97 Registered Member

    Joined:
    May 6, 2009
    Messages:
    71
    Likes Received:
    7
    I agree, it would have to be designed in a way to delete any poor performing proxies and only use the ones which are active. I'm actually going to try and slow it down a bit and see, but I think they're blocking based on the quantity and number of request coming from a single IP, but I'll do a test to see. Do you have any good scraping proxies which are cheaper than crawlera?
     
  7. indeed97

    indeed97 Registered Member

    Joined:
    May 6, 2009
    Messages:
    71
    Likes Received:
    7
    It's a bot I had developed for personal use.
     
  8. Chris.Roark

    Chris.Roark Jr. VIP Jr. VIP

    Joined:
    Aug 16, 2016
    Messages:
    1,519
    Likes Received:
    511
    Home Page:
    Why not you "tame" it down a bit and slow your scraping speed. You might scrape fewer data/pages, but in the long run you might end up scraping more.
     
  9. indeed97

    indeed97 Registered Member

    Joined:
    May 6, 2009
    Messages:
    71
    Likes Received:
    7
    Yea, I can do that, but I'm looking to perform 1-2M requests a day. Even if I tame it down to avoid the bans, I need additional scripts running in parallel to get me greater return in a shorter time span. So I'm thinking of some alternative methods which involve alternative proxies.