Hi Guys
I'm hoping someone can help me out please, I have purchased the Scrapebox Premium Email Scraper Plugin and I want it to scrape emails from the 'crawl loaded list' option.
I have a list of 1700 domains to scrape, when I set the options to:
worker threads - 100
Level 1 - I assume this means it only crawls the one site, or is it just the 1 page of the URL in the loaded list ?
3 emails per domain.
Then after completing the crawl it has returned just 120 emails from a list of 1700.
Then if I set it to
worker threads - 100
Level 2 - I assume this means it crawls deeper into the site, or does it follow links to other sites, as sometimes it says site in que have been in excess of 11000 sometimes.
3 emails per domain.
When I have run it at the above settings, it has literally run for hours, the longest I have left it to run is 11 hrs, and it still had 2000 plus site in the que to run.
Then when I have stopped the program as above it freezes and doesn't let me export the email collected.
I have attached a screenshot that I stopped after just 5 mins, and it doesn't let me export the emails, it just freezes after I press stop.
If someone could help me out please for recommendations for crawling large lists effectively.
I have watched all of the tutorial videos, any help would be gladly received.
Do I need to use proxies? Would that help?
Cheers
I'm hoping someone can help me out please, I have purchased the Scrapebox Premium Email Scraper Plugin and I want it to scrape emails from the 'crawl loaded list' option.
I have a list of 1700 domains to scrape, when I set the options to:
worker threads - 100
Level 1 - I assume this means it only crawls the one site, or is it just the 1 page of the URL in the loaded list ?
3 emails per domain.
Then after completing the crawl it has returned just 120 emails from a list of 1700.
Then if I set it to
worker threads - 100
Level 2 - I assume this means it crawls deeper into the site, or does it follow links to other sites, as sometimes it says site in que have been in excess of 11000 sometimes.
3 emails per domain.
When I have run it at the above settings, it has literally run for hours, the longest I have left it to run is 11 hrs, and it still had 2000 plus site in the que to run.
Then when I have stopped the program as above it freezes and doesn't let me export the email collected.
I have attached a screenshot that I stopped after just 5 mins, and it doesn't let me export the emails, it just freezes after I press stop.
If someone could help me out please for recommendations for crawling large lists effectively.
I have watched all of the tutorial videos, any help would be gladly received.
Do I need to use proxies? Would that help?
Cheers