Need help scraping with proxies?

N

notmyrealnameoryear88

Newbie
Joined
Jul 17, 2023
Messages
14
Reaction score
0
I'm trying to scrape a website a few million pages, and I can do it locally with 0 issue no rate limiting after a week and all, but i've been looking around and see a lot about getting residential IPs for this task. However, I see that theres no real static residential IP's so I'm not really sure how it works and i was hoping someone could possibly give me a bit of a hand? Right now I have a program scraping the links using no proxy just my actual IP, and i have a secondary that will use a proxy file filled with proxies( just a txt) and cycle them. Any public IP lists are automatically rate limited which is why im perfectly fine getting a paid solution I just feel like if i just find something I'll be scammed 11 times before I find one that works. Aswell as if its a rotating service is it more like get a 429 and make a new request for a proxy through the API type thing? Sorry if this is a stupid question I'm just trying to do a fun project. Thank you very much guys!
 
Advertise on BHW
I would definitely try rotating residential proxies. You get access to a huge network of IP's that look like real users and you only pay for the data you consume
 
I'm good at doing Scraping task as we are doing G scraping since years so i might help you if you need.

Most of the time Semi Dedicated and Dedicated proxies are good enough for scraping tasks.

Static Residential Proxies are good but little more expensive then Semi Ded or Dedicated data center proxies.

It all depends on what site you are scraping.

If you can give details about that site i might help you .
 
extremeboy said:
I'm good at doing Scraping task as we are doing G scraping since years so i might help you if you need.

Most of the time Semi Dedicated and Dedicated proxies are good enough for scraping tasks.

Static Residential Proxies are good but little more expensive then Semi Ded or Dedicated data center proxies.

It all depends on what site you are scraping.

If you can give details about that site i might help you .
Click to expand...
I'm scraping spotify, my personal IP only gets 1 429 per like day and i sleep for 10 seconds and all set. i do 4 "scrapes" per 2 seconds. the payload per page is ~ 100KB

I should note. Data center proxies seem to get 429'd almost instantly.
 
Last edited:
Have you thought of using a vps? This would free up your PC and one instance can provide thousands of ipv6 addresses. Cloud solutions can be super cheap
 
notmyrealnameoryear88 said:
I'm scraping spotify, my personal IP only gets 1 429 per like day and i sleep for 10 seconds and all set. i do 4 "scrapes" per 2 seconds. the payload per page is ~ 100KB

I should note. Data center proxies seem to get 429'd almost instantly.
Click to expand...
Because they have security in place otherwise any one abuse their site any time and get platform down.

You have to tackle it.
 
A single rotating residential or mobile proxy per scraper instance should work for you!

You could either rotate IP on a schedule or automatically if / when rate limited via the proxy provider's API. This will land you on a new IP and should allow you to keep scraping.

Check out the Proxies For Sale section and take out some free trials of a few provider's proxies to see how they perform for your usage.
 
Look into Zyte Smart Proxy Management. I used their service 1-2 years ago.

They have a trial for 30 days/10gb, you can basically make countless accounts and have as much gb as you want.
 
Dodge the public proxies entirely. You'll almost always get unreliable IPs on them. Whether it's mobile or residential proxies depends on the specifics of your setup, the important bit is you're getting rotating residential IPs for scraping.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top