yellowpages.com plugin - freezes

Hi,

I'm new to Scrapebox and running into an odd issue trying to use the yellowpages.com plugin. I'm trying to do some test runs so
a) I harvested tested and picked up anon proxies
b) added about 100 keywords
c) picked about 100 locations and started the scraping
d) left the timeout / retries settings to default (0 proxy retires and 10 for timeout)

After a couple hrs it appears that it had scraped some 6000 records for just 1 location, and the activity appears to stop - although the only button enabled is "stop". I waited for a bunch and finally clicked stop - but that doesn't do anything either. Now even though only 6k records were scraped - I can't still export or save them ... I had to kill scrapebox and start over

Any suggestions pls?

Thanks much!

Oto Scraper said:

Hi,

I'm new to Scrapebox and running into an odd issue trying to use the yellowpages.com plugin. I'm trying to do some test runs so
a) I harvested tested and picked up anon proxies
b) added about 100 keywords
c) picked about 100 locations and started the scraping
d) left the timeout / retries settings to default (0 proxy retires and 10 for timeout)

After a couple hrs it appears that it had scraped some 6000 records for just 1 location, and the activity appears to stop - although the only button enabled is "stop". I waited for a bunch and finally clicked stop - but that doesn't do anything either. Now even though only 6k records were scraped - I can't still export or save them ... I had to kill scrapebox and start over

Any suggestions pls?

Thanks much!

Click to expand...

If you are going to use scraped proxies, then you want to build a custom test for the yellow pages your using as the lions share of them are probably blocked by yellow pages, or has been my experience anyway.


Further you want to max out that proxy retries.


Also Im not sure how you loaded in 100 locations and 100 keywords, that makes no sense. Are you on windows or mac?

All results are auto saved inside the plugin folder, so scrapebox folder \ plugins \ ypscraper \ auto save

so they are not lost.

But your not being able to click stop is a thread locked issue. so here is some info on that



That means that something has locked 1 or more of the threads. This can be security software such as anti-virus, malware checkers and firewalls. So you should whitelist scrapebox in all security software and then you can whitelist the entire scrapebox folder as well.



Further any program that accesses the internet can lock threads, things like skype, utorrent etc… So you can try closing down any unneeded programs. Then if its working you can turn programs back on 1 by 1 to find the culprit.



Further computer optimization software can lock threads so you can shut any such software down.



Take note that disabling security software (such as anti-virus, malware checkers and firewalls) often only stops new rules form forming, but allows existing rules to still fire. So you have to fully whitelist in the security software or uninstall the security software(as a test).



Further some security softwar requires you to whitelist in more then one place before it takes effect.



Also note that disabling a router firewall, does actually fully disable it.





Basically you have to sort out what is locking the threads, because scrapebox is forced to wait until all threads are released. On occasion it can be your operating system that does it, so you can try restarting your machine and/or lowering total connections.

loopline said:

If you are going to use scraped proxies, then you want to build a custom test for the yellow pages your using as the lions share of them are probably blocked by yellow pages, or has been my experience anyway.

Click to expand...

Thanks much Loopline for the detailed help! I will try these out.
Best regards

Oto Scraper said:

Thanks much Loopline for the detailed help! I will try these out.
Best regards

Click to expand...

Your welcome, have a great day!

loopline said:

Your welcome, have a great day!

Click to expand...

Thanks much again for the pointers Loopline and I was able to make some progress. I got a run where I was able to get 35k records . I noticed one thing and trying to see if there is a way to address this.

I started with 40 connections and noticed that over time, the connection count keeps decreasing. I set proxy retries to 5, timeout 10 and delay 0. So I've been waiting for it to get close to 0. stop and then remove the locations that have been completed and start again. This will need me to be sitting in front of this screen, and I'm trying to see if I could give it a big load to work thru the night .

I've got some 200 proxies and have 50 locations and 300 keywords to work thru.

Thanks much in advance

I found a couple more issues ...
a) I can't find the autosave folder under plugins/YellowPage Scraper ... so looks like nothing is being saved - is there a setting to enable autosave pls?
b) is there a way to close or release these stuck connections without killing scrapbox itself - so I can atleast try and salvage the scraping that has been complete??

I'm using a Mac - v10.14.2

Thanks much!

Oto Scraper said:

I found a couple more issues ...
a) I can't find the autosave folder under plugins/YellowPage Scraper ... so looks like nothing is being saved - is there a setting to enable autosave pls?
b) is there a way to close or release these stuck connections without killing scrapbox itself - so I can atleast try and salvage the scraping that has been complete??

I'm using a Mac - v10.14.2

Thanks much!

Click to expand...

Anyone that could help pls?? I keep getting stuck and having to restart again and again ...

BTW - on my install there is no Autosave folder in the scrapebox/plugins/yellowpage scraper/ path - there are just 3 txt files about the yellow page plugin setting and locations it appears ...

@loopline proxy test module is not working for Mac.

Oto Scraper said:

Thanks much again for the pointers Loopline and I was able to make some progress. I got a run where I was able to get 35k records . I noticed one thing and trying to see if there is a way to address this.

I started with 40 connections and noticed that over time, the connection count keeps decreasing. I set proxy retries to 5, timeout 10 and delay 0. So I've been waiting for it to get close to 0. stop and then remove the locations that have been completed and start again. This will need me to be sitting in front of this screen, and I'm trying to see if I could give it a big load to work thru the night .

I've got some 200 proxies and have 50 locations and 300 keywords to work thru.

Thanks much in advance

Click to expand...

Its going down because either something is locking threads or its running out of proxies. Tyr putting your proxy retries to the max.

Oto Scraper said:

I found a couple more issues ...
a) I can't find the autosave folder under plugins/YellowPage Scraper ... so looks like nothing is being saved - is there a setting to enable autosave pls?
b) is there a way to close or release these stuck connections without killing scrapbox itself - so I can atleast try and salvage the scraping that has been complete??

I'm using a Mac - v10.14.2

Thanks much!

Click to expand...

You can't release the threads, scrapebox must wait on mac to release it and mac isn't releasing them either because mac has locked the threads or something else on your machine has locked them.

The auto save may not be something built into mac, Ill have to double check, I know its on windows.

mrlucifer7 said:

@loopline proxy test module is not working for Mac.

Click to expand...

Can you be more specific? What doesn't work? It works fine for me. Is it giving you an error?

loopline said:

Its going down because either something is locking threads or its running out of proxies. Tyr putting your proxy retries to the max.





You can't release the threads, scrapebox must wait on mac to release it and mac isn't releasing them either because mac has locked the threads or something else on your machine has locked them.

The auto save may not be something built into mac, Ill have to double check, I know its on windows.





Can you be more specific? What doesn't work? It works fine for me. Is it giving you an error?

Click to expand...

Hey @loopline I sent you a couple DM's on instagram inquiring about some of your services regarding contact marketing since I can't send PM's on here.

@loopline
I am seeing “Processing...” In the following part connection always shows 0. Is there any update coming soon? It is very important module for me.

Oto Scraper said:

I found a couple more issues ...
a) I can't find the autosave folder under plugins/YellowPage Scraper ... so looks like nothing is being saved - is there a setting to enable autosave pls?
b) is there a way to close or release these stuck connections without killing scrapbox itself - so I can atleast try and salvage the scraping that has been complete??

I'm using a Mac - v10.14.2

Thanks much!

Click to expand...

So mac did not have the auto save feature but scrapebox added it. so if you update to the new mac version you should be good on that.

Terminator199 said:

Hey @loopline I sent you a couple DM's on instagram inquiring about some of your services regarding contact marketing since I can't send PM's on here.

Click to expand...

Thanks, I replied.

mrlucifer7 said:

@loopline
I am seeing “Processing...” In the following part connection always shows 0. Is there any update coming soon? It is very important module for me.

Click to expand...

It works for me. Has it always done this for you? Are you updated to the latest version?

It is likely something on your machine or network messing with the connections. Does it eventually complete or ?

@loopline I’m using the latest version. From begining to today is not working on my mac. I turned off even the security wall. Also my network is working well. Because my windows pc is using the same network and it is working on it. However i just want to mac so it is very important module for me. How can i fix it. Last but not least proccessing never finish

mrlucifer7 said:

@loopline I’m using the latest version. From begining to today is not working on my mac. I turned off even the security wall. Also my network is working well. Because my windows pc is using the same network and it is working on it. However i just want to mac so it is very important module for me. How can i fix it. Last but not least proccessing never finish

Click to expand...

Is the proxy responding? I mean you have a windows version of scrapebox yes? If you put the same proxies in the windows version does it test?

I did try it on mac and it works fine for me. Ive also not heard of any other mac issues having an issue like this with proxy tester, ever. I mean the scrapebox code for proxy testing on mac is working for a few thousand mac users, myself and for scraprbox support. So its something on your mac, outiside of scrapebox or something in the network. I can't say with certainty what that might be though. You can make sure you whitelist in any security software you may have and shut down any non crucial programs as a test. Then you can turn them back on 1 by 1 till you find the culprit, assuming that works.

loopline said:

So mac did not have the auto save feature but scrapebox added it. so if you update to the new mac version you should be good on that.

Click to expand...

I updated to the latest ver and there is an autosave folder on mac now - but incidentally I noticed that the file being saved is saved in the plugins folder itself - as it uses the "\" in the file name ie autosave\<filename> ... - I minor bug I guess

now I can atleast kill scrapebox and not loose all of the scraping job .

BTW - is there a way we can load the scrape results into the plugin so we can do the dup/filter stuff ??

thanks