Need help scraping with proxies?

[notmyrealnameoryear88]

I'm trying to scrape a website a few million pages, and I can do it locally with 0 issue no rate limiting after a week and all, but i've been looking around and see a lot about getting residential IPs for this task. However, I see that theres no real static residential IP's so I'm not really sure how it works and i was hoping someone could possibly give me a bit of a hand? Right now I have a program scraping the links using no proxy just my actual IP, and i have a secondary that will use a proxy file filled with proxies( just a txt) and cycle them. Any public IP lists are automatically rate limited which is why im perfectly fine getting a paid solution I just feel like if i just find something I'll be scammed 11 times before I find one that works. Aswell as if its a rotating service is it more like get a 429 and make a new request for a proxy through the API type thing? Sorry if this is a stupid question I'm just trying to do a fun project. Thank you very much guys!

 

[Proxy Cake]

I would definitely try rotating residential proxies. You get access to a huge network of IP's that look like real users and you only pay for the data you consume

 

[extremeboy]

I'm good at doing Scraping task as we are doing G scraping since years so i might help you if you need.

Most of the time Semi Dedicated and Dedicated proxies are good enough for scraping tasks.

Static Residential Proxies are good but little more expensive then Semi Ded or Dedicated data center proxies.

It all depends on what site you are scraping.

If you can give details about that site i might help you .

 

[notmyrealnameoryear88]

I'm good at doing Scraping task as we are doing G scraping since years so i might help you if you need.

Most of the time Semi Dedicated and Dedicated proxies are good enough for scraping tasks.

Static Residential Proxies are good but little more expensive then Semi Ded or Dedicated data center proxies.

It all depends on what site you are scraping.

If you can give details about that site i might help you .

I'm scraping spotify, my personal IP only gets 1 429 per like day and i sleep for 10 seconds and all set. i do 4 "scrapes" per 2 seconds. the payload per page is ~ 100KB

I should note. Data center proxies seem to get 429'd almost instantly.

 

[derregenrennt]

Have you thought of using a vps? This would free up your PC and one instance can provide thousands of ipv6 addresses. Cloud solutions can be super cheap

 

[HostMeNow]

Use rotating proxy's or a large list of semi private proxy's. If you use say 100, scraping will be easy.

 

[RamzaC1]

Use rotating proxy's or a large list of semi private proxy's. If you use say 100, scraping will be easy.

Semi private proxy's?

 

[turelink]

do not waste time on public proxies,
here is a new project for scraping, hope can help you,
https://www.gptsapp.io/gpts-hunting-feed

 

[extremeboy]

I'm scraping spotify, my personal IP only gets 1 429 per like day and i sleep for 10 seconds and all set. i do 4 "scrapes" per 2 seconds. the payload per page is ~ 100KB

I should note. Data center proxies seem to get 429'd almost instantly.

Because they have security in place otherwise any one abuse their site any time and get platform down.

You have to tackle it.

 

[proxygo]

i remember years ago with scrapebox when Goggle tightened
the screws on scraping you needed to have your thread count
very low to avoid ip bans

 

[PrestigeUK]

A single rotating residential or mobile proxy per scraper instance should work for you!

You could either rotate IP on a schedule or automatically if / when rate limited via the proxy provider's API. This will land you on a new IP and should allow you to keep scraping.

Check out the Proxies For Sale section and take out some free trials of a few provider's proxies to see how they perform for your usage.

 

[Cybersinner]

Look into Zyte Smart Proxy Management. I used their service 1-2 years ago.

They have a trial for 30 days/10gb, you can basically make countless accounts and have as much gb as you want.

 

[TechTrekTales]

How do you scrape data coding or extensions

 

[imccafey]

Dodge the public proxies entirely. You'll almost always get unreliable IPs on them. Whether it's mobile or residential proxies depends on the specifics of your setup, the important bit is you're getting rotating residential IPs for scraping.

 

[proxygo]

do not waste time on public proxies,
here is a new project for scraping, hope can help you,
https://www.gptsapp.io/gpts-hunting-feed

public proxys only worked for scraping between 2005 - 2018
then with all the blocks, that freebie died