I need proxies for yellowpages.com.au

[eric5037]

Hi,

I need proxies to scrap data from yellowpages.com.au.

Those proxies must be:
- Elite / premium / highly anonymous proxies
- Private / dedicated proxies
- Residential proxies
- Autorotating proxies on each request

Right now, I have "residential rotating proxies" in stormproxies but the rotation happens every 5 minutes and I need it to happen on each request.

Months ago, I had "dedicated rotating proxies" and the rotation happens on each request but they're not all "residential proxies", there are "datacenter proxies" too

I know there is a section dedicated to proxies so I've been in each one of them but I'm kind of lost in so many offers so may be someone with experience can tell me what the best proxies are. I think I can pay around 50$ / month.

Thanks in advance,

 

[devkareem]

ProxyMesh rotates on every request. I've tried them before for scraping google and went fine. You could try the free trail first.

 

[eric5037]

ProxyMesh rotates on every request. I've tried them before for scraping google and went fine. You could try the free trail first.

Thanks but what I read is that:
"ProxyMesh provides 15 rotating IP address proxy servers, each with 10 IP addresses rotating twice per day, giving you a total of 300 IP addresses per day. When you make a request thru one of these 15 anonymous proxy servers, your request will be randomly routed through one of 10 different proxy IP servers. This make it much harder for sites to track you based on IP addresses. And each location changes IP addresses twice a day, to help you get around IP bans or avoid IP rate limits. All without any special software to download or manual proxy switching. Read more about ProxyMesh https://www.proxymesh.com/features/, https://www.proxymesh.com/pricing/, https://www.proxymesh.com/faq/, or try it out for free with our https://www.proxymesh.com/web/index.php. You can also https://www.proxymesh.com/account/signup/free/?placement=index. There are no bandwidth limits for paying accounts, and you can cancel at any time."

So I don't see that it rotates on each request. Am I not in the right section?

 

[Chris.Roark]

I think I can pay around 50$ / month.

I don't think you will find real residential IPs for this price, that's why you got residential mixed with datacenter ones.

For the moment, residential proxies are quite expensive

 

[devkareem]

Thanks but what I read is that:
"ProxyMesh provides 15 rotating IP address proxy servers, each with 10 IP addresses rotating twice per day, giving you a total of 300 IP addresses per day. When you make a request thru one of these 15 anonymous proxy servers, your request will be randomly routed through one of 10 different proxy IP servers. This make it much harder for sites to track you based on IP addresses. And each location changes IP addresses twice a day, to help you get around IP bans or avoid IP rate limits. All without any special software to download or manual proxy switching. Read more about ProxyMesh Features, Pricing Plans, FAQ, or try it out for free with our anonymous web proxy. You can also signup for a free trial. There are no bandwidth limits for paying accounts, and you can cancel at any time."

So I don't see that it rotates on each request. Am I not in the right section?

There is no service can offer unlimited IPs. I think 300 a day is good enough. On every request it selects randomly one of the 300 IPs which changes frequently too.

In this way, every request goes through a random IP address choosen from among the current pool of IP addresses assigned to forward proxy servers. This pool periodically changes, when new proxy servers are added, and old proxy servers are removed, thus changing IP addresses periodically. With rotating IP addresses, IP bans and rate limiting will be a thing of the past.

 

[lokiys]

You did not mention what is amount of data You want to scrape per hour, day, week ?
Thats the biggest question. To suggest anything.

 

[eric5037]

I don't think you will find real residential IPs for this price, that's why you got residential mixed with datacenter ones.

For the moment, residential proxies are quite expensive

Everything is always too expensive

Now... this is what I read in your quite interesting site:
"They're not always that expensive. You'll surely be able to find cheap residential proxies if you make an effort."

I will make the effort to keep looking for it. I have to say that I am surprised about the proxy market because I have only identified may be 60 proxy service providers which seems very few of them to me. Am I missing something?

 

[eric5037]

There is no service can offer unlimited IPs. I think 300 a day is good enough. On every request it selects randomly one of the 300 IPs which changes frequently too.

I think the solution may be here: the size of the pool of proxies.

The problem is that I don't really know how to calculate/observe that. Right now, I have "backconnect residential proxies" from stormproxies but after a good while, I keep being banned. Last time I checked, I was able to make 600 connects but I don't even know how many times I have changed my IP. I'm kind of lost here.

What I see in all those proxy service providers are a certain amount of offers but I think that the best is to talk with and I would almost say "negotiate" with them. I am in talks with privateproxyme.

 

[eric5037]

You did not mention what is amount of data You want to scrape per hour, day, week ?
Thats the biggest question. To suggest anything.

you're absolutely right:

I'm thinking about:
1 request / 5 seconds => 720 / hour => 17280 / day

To give you the big picture of what I'm trying to achieve:
I want all the data for the whole country (Australia) for certain keywords. The problem is that the australian yellow pages are limited to 29 pages so If have less than that amount, there is not a single problem but if I have more, I will have to narrow my search by cities. Is that a problem? well, there are more than 15 000 cities in Australia. With 17280 / day (and not even thinking in pagination), 1 day of extraction = 1 keyword.

 

[eric5037]

what about this:
https://dreamproxies.com/

 

[lokiys]

you're absolutely right:

I'm thinking about:
1 request / 5 seconds => 720 / hour => 17280 / day

To give you the big picture of what I'm trying to achieve:
I want all the data for the whole country (Australia) for certain keywords. The problem is that the australian yellow pages are limited to 29 pages so If have less than that amount, there is not a single problem but if I have more, I will have to narrow my search by cities. Is that a problem? well, there are more than 15 000 cities in Australia. With 17280 / day (and not even thinking in pagination), 1 day of extraction = 1 keyword.

Well then I suggest to buy higher plan from stormproxies, like more ports so You can scrape in multithreaded way.
I'm not sure how do You scrape all of that. But if amount is such big then residental proxies would be the best way to go.
Maybe You also want to order some quality bot for your scrapping. Solely by your requirements.

Because for such big amount of data You will also need some database, because managing such amount is very complex if You do not have own databases. But thats other topic.