- Jan 25, 2009
- 6,378
- 3,774
Did the recent update allow SB to store more than 1mil urls in URLS's harvested list? I left it scraping for an hour and when I returned I had 1.2mil results.
Also,I haven't been really scraping heavily but I've begun crashing now. If I leave it too long, it may crash. If I load too many keywords, it also crashes when I click stop harvesting or when it finishes but i fixed that by splitting up my keywords list.
So how much RAM would I need per connections to keep it stable during harvesting? 1GB per 50? Thanks Sweetfunny.
No Microsoft is actually the one that sets the limit of 1 million for that style of grid. Just sometimes after you finish harvesting scrapebox will report it has more urls in there then 1 mil. It saves them off in the harvested sessions folder.
I don't really have a set amount, but with 8GB I harvested 140 million urls in one run, which took a solid week and no problems.