Using ScrapeBox to get 500k+ emails from 1 site

[NLHES]

Hi - I'm trying to scrape a huge online forum for email addresses that people have posted. I did one test with 10k proxies, it worked really well but stopped on the 9th thousand email or so. I had to force close ScrapeBox to get out of the operation.

Anyways - based on number of pages in this forum at question, combined with the % of pages with an email address based on the 8.5k or so emails I got, I was able to estimate there are around 500k-700k emails on this forum that I'd like to have.

Two questions:

- what could of been the error that I encountered?

- what's the best way to go for such a huge operation / email list? Like, how many proxies shall I use, obv it will take time also - shall I set up another PC to do this? Like - has anyone ever done anything like this and can give generic advice / things to watch out on such a big email list?


Thanks all!

 

[rocks11111]

following

 

[loopline]

WEll if you could get a list of all the urls, or better if they follow a patter just fabricate them. Then you can break it up into chunks of urls and work with them. That way if something happens and there is a crash etc.. then your not stuck.

Also I assume your using private proxies? If public that was probably the issue because it sounds like locked threads was your issue. So make sure you whitelist scrapebox in all security software such as anti-virus, malware checkers and firewalls.

If you have chunks of urls then you can run multiple instances of scrapebox on 1 machine, (assuming your using windows) and put like 2 private proxies in each instance, or 3 proxies if you want and then just set connections to 2 and let it go. Depending on the site, you might actually need 4 or 5 proxies per instance with 2 connections.

OR just turn scraperbox loose on 1 connection with no proxies and see what happens. it will go faster then you think with no proxies, and then proxies can't be a choke point for locked threads either.

 

[whiteogreguy]

infinite loop or server banned you for too many requests too fast.

 

[gloriousbach]

Sometimes with really heavy operations scrapebox crashes or hangs

 

[loopline]

Sometimes with really heavy operations scrapebox crashes or hangs

Yes, a 64bit version of Scrapebox is going to be better for heavy operations, because if you run out of memory due to 32bit or if you just run out of resources on your machine etc... it can cause a crash. So a machine that has the resources you need is crucial and 64bit windows and 64bit scrapebox is also ideal.

Further you can just break things up into chunks and work with them.

 

[NLHES]

Thanks all for the replies. I think the idea of getting all the links and doing it chunk-by-chunk makes sense, the only problem...the site has 4,4mn pages, according to Google.

So how would I go for getting the list of the 4,4mn? I'm expecting a crash even for that..is there a way I can break this process down too and go chunk by chunk?

Also - what's the best way / instruction for getting the links? I know Link Extractor would be the best addon, but it's grey in the addon list and I'm unable to install it?

Lastly - do you guys think ScrapeBox might be out of reach for such a big task? Shall I look for other alternatives, possibly more custom solutions?

 

[salamsoosis]

Yes, a 64bit version of Scrapebox is going to be better for heavy operations, because if you run out of memory due to 32bit or if you just run out of resources on your machine etc... it can cause a crash. So a machine that has the resources you need is crucial and 64bit windows and 64bit scrapebox is also ideal.

Further you can just break things up into chunks and work with them.

Would you please message me? Thanks!