need a bulk website checker

phil stewart

Regular Member
Joined
Feb 20, 2019
Messages
348
Reaction score
109
I have a massive list of urls I want to check and take out the dead ones. I have tried with scrapebox alive check but it's waaaay too slow and only goes up to 200 threads. Anyone got ideas for what I can use instead?
 
You can run an unlimited number of scrapebox instances on windows. so assuming your not using mac, then just setup 5 instances and you can run 1000 threads.

windows does not "like" it when you run more then 100 to 200 threads per instance of a program. In the past scrapebox did much more threads, but it causes way more problems then it solves. Even google uses lots of smaller machines rather then fewer larger ones.

Plus if 1 crashes the rest are still running. Its far better to just divide it up, its more efficient, it provides crash protection and it works much better with windows, although 1000 to 2000 threads is the upper limit for windows being able to handle it for most setups.
 
Back
Top