Scrapebox - yellowpage scraper - rotating proxies

[Oto Scraper]

Hi, I'm trying to scrape some yellow pages using the yellowpage plugin in scrapebox. I first tried harvesting some proxies, but was having a bunch of issues since the connections kept freezing or scrapebox was constantly dropping them.

So I saw the article on back connect proxy and signed up for a 80 thread package with storm proxies. Unfortunately, I still can't seem to get the jobs to run for some time, every 30-45 mins the connections go down to zero, although there are still many locations to do.

I've tried both their US/EU proxies as well as their worldwide (best for scraping) options and although yes both do work, in the US EU case I've never been able to run for more than an hour before the connections are down to a single digit and the it crawls to scrape . With the worldwide one, although I was able to get a job to run for 7 hrs before the connections came down to single digits, the amt that was scraped was less than the us/eu 1 hr job .

I've reached out to storm proxies, but don't seem to have any good suggestions.

I've added my IP to the stormprox dashboard as "an authorized ip" and taken their "main gateway" ips and put them in the proxy in the yellowpage plugin.

Any thoughts on any tweaks to get my to get these jobs going.

Thanks much in advance!

 

[loopline]

In the yellow pages plugin you want to max out the proxy retries and timeouts. This will give you the most possible chances.

Then you need to duplicate your gateway IP at least 10 times, because you will have 10 retries, but like 15 is better, as its a bit random.

So if your gateway IP is

123.123.123.112:8080

Then you would do

123.123.123.112:8080
123.123.123.112:8080
123.123.123.112:8080
123.123.123.112:8080
123.123.123.112:8080
123.123.123.112:8080
123.123.123.112:8080
123.123.123.112:8080
etc...

15 times.

Then try it and see how it goes. After the retries and connections and proxies are exhausted, the plugin will eventually quit. If you get done with everything, great, if not, then you can just divide your keywords or locations into chunks and run each chunk.

 

[SoccerLover]

THREAD MOVED​

 

[Oto Scraper]

In the yellow pages plugin you want to max out the proxy retries and timeouts. This will give you the most possible chances.

Then you need to duplicate your gateway IP at least 10 times, because you will have 10 retries, but like 15 is better, as its a bit random.

15 times.

Then try it and see how it goes. After the retries and connections and proxies are exhausted, the plugin will eventually quit. If you get done with everything, great, if not, then you can just divide your keywords or locations into chunks and run each chunk.

Thanks much loopline - will try this.

 

[loopline]

Thanks much loopline - will try this.

Your welcome, have a great day!

 

[kveldulv]

Proxy retries makes a huge difference when scraping yellow pages.

Also don't go overboard on the threads, with 80 proxies, using 20 threads, they will still get banned quite quickly.
Yellowpages are super aggressive with banning bots and scrapers.

 

[seodestruction]

I cant get mine to even show me results it stays on 0/15998 and then after like 10 min I press stop it shows me like 4 leads with unfinished phone numbers, I put my main gate backwards proxie from storm proxie I can harvest fine with harvester tool with the same proxie but not with my newly bought yellowpages plugin?

 

[kveldulv]

I cant get mine to even show me results it stays on 0/15998 and then after like 10 min I press stop it shows me like 4 leads with unfinished phone numbers, I put my main gate backwards proxie from storm proxie I can harvest fine with harvester tool with the same proxie but not with my newly bought yellowpages plugin?

Try opening yellowpages with those proxies set in your browser and doing the anti-bot test.

 

[seodestruction]

Now I got it to work but its super slow mind you I only have 40 threads this is my settings can yo u help?

 

[loopline]

I cant get mine to even show me results it stays on 0/15998 and then after like 10 min I press stop it shows me like 4 leads with unfinished phone numbers, I put my main gate backwards proxie from storm proxie I can harvest fine with harvester tool with the same proxie but not with my newly bought yellowpages plugin?

copy and paste your gateway ip in the proxies box over and over, at least 10 times, if not 20. It may not make sense, but scrapbox sees each line as unique proxy so then it will retry and its changing the ip on teh back end so it will work better.

 

[seodestruction]

you're a legend to me LoopLine and I have been watching your videos daily back to back over and over again as my scraping bible, its been working awesomely!

 

[seodestruction]

One last thing, On YP SCRAPER im using only 40 threads but of course I can only use the 25% of it on harvesting so technically im using 10, is that the reason why its soooo slow? its like paintfully slow I get 1/37 location results after like 5 whole min with only 10 Kw's per location! Am I doing something wrong? or is this outcome to be expected with the $39.99 a month package?

 

[loopline]

you're a legend to me LoopLine and I have been watching your videos daily back to back over and over again as my scraping bible, its been working awesomely!

Glad they are of a help, thanks for the kind words.

One last thing, On YP SCRAPER im using only 40 threads but of course I can only use the 25% of it on harvesting so technically im using 10, is that the reason why its soooo slow? its like paintfully slow I get 1/37 location results after like 5 whole min with only 10 Kw's per location! Am I doing something wrong? or is this outcome to be expected with the $39.99 a month package?

The ypscraper is 1 thread per keyword. So if you are using many locations and only 1 keyword, then it should be using only 1 thread.
Yellow pages keeps the info on internal pages, so every single location has its own page and it must all be loaded. Yellow pages does a lot to prevent its data from being scraped, so sometimes its just a bit slow.

Try different connection counts and see if there is variance in the speed.

 

[Sako94]

Hi guys and @loopline

Im having some issues getting scrapebox to work with YellowPages as well.

I bought backconnect 10 thread proxies from Stormproxies.

Currently I have these as my settings and what I dont understand why every record that ive scraped which shows 16 of them all have blank lines for every result, what am I doing wrong?

I appreciate the feedback guys.

 

[loopline]

Hi guys and @loopline

Im having some issues getting scrapebox to work with YellowPages as well.

I bought backconnect 10 thread proxies from Stormproxies.

Currently I have these as my settings and what I dont understand why every record that ive scraped which shows 16 of them all have blank lines for every result, what am I doing wrong?

I appreciate the feedback guys.

Are you using the latest version of Scrapebox for mac?

What happens if you turn off the proxies and try a sample scrape, does it work?

 

[syedamaham]

I have a yellow page scraper. No need of any proxy, can scrap thousands of emails with details from yellowpages website. I can give you that.

 

[MaestroDelWeb]

@loopline I'm having an issue with YP scraping also. Hope you can help. I'm using over 1000 semi-private proxies. I set it to run on 40 connections (had it on 20 first but got the same results). It only shows 5 connections, then it gets to 245 scrapes and the connections stop.

This is what it looks like when I'm starting off. There is no way for me to change delay. It's black and won't slide.

This is what it looks like after it stops. It only does one keyword out of the 5.

I've used it plenty of times with no issues. I'm not sure what could be wrong.

Any help or info would be appreciated.

 

[loopline]

@loopline I'm having an issue with YP scraping also. Hope you can help. I'm using over 1000 semi-private proxies. I set it to run on 40 connections (had it on 20 first but got the same results). It only shows 5 connections, then it gets to 245 scrapes and the connections stop.

This is what it looks like when I'm starting off. There is no way for me to change delay. It's black and won't slide.

This is what it looks like after it stops. It only does one keyword out of the 5.

I've used it plenty of times with no issues. I'm not sure what could be wrong.

Any help or info would be appreciated.

Its 1 connection per keyword. So you have only 5 keywords so its only going to use 5 connections. If you had 437 keywords (or any number equal to or greater then 40) it would use 40 connections.

The delay only applies if you are using 1 connection. Because if you need more then 1 connection then you don't need a delay, because the same end speed can be achieved with just lowering connections rather then a delay and thats more recourse effecient then trying to make your machine manage a delay on 40 different threads.

As for only scraping 1 keyword, try a sample with no proxies and 1 connection, does it do more then 1 keyword?

I would up your timeouts a bit too, maybe even max them, just for a test.

most generally speaking it only stops when it runs out of proxies it knows are good.

So even though there are 10 proxy retries, if it encounters 10 bad proxies then it leaves off of that keyword.

First try this, build a custom test and test your proxies against yellow pages

Then only load in ones that pass, does that make it work?

Worst case we can go down the route of hacking the ini and running the retries way up, but thats exceedingly ineffecient and massively slows it down, just testing the proxies is the most ideal solution.

 

[MaestroDelWeb]

Ok. I thought the 10 proxy retries was 10 retries per proxy, I didn't realize it was 10 proxies per keyword. I will definitely test them against YP. I thought because of the quantity of proxies there was no way to fail. Thanks for the advice definitely gives me something to make everything more efficient.

 

[loopline]

Ok. I thought the 10 proxy retries was 10 retries per proxy, I didn't realize it was 10 proxies per keyword. I will definitely test them against YP. I thought because of the quantity of proxies there was no way to fail. Thanks for the advice definitely gives me something to make everything more efficient.

It is 10 retries, but its not going to retry a proxy that is blocked, that would be just 10 retries of the same blocked proxy which would be futile.

Each retry is a different proxy.

The qualify of the proxy has no bearing on whether or not yellow pages will block the ip. An IP is an IP to yellow pages. So Im not saying its a proxy quality issue Im saying most likely some or many if not almost all of your proxies are blocked by yellow pages, hence the custom test will sort that for you.