distributed

I know crawling interest a lot of people so I decided to post a small example of how to scale it and what tech to use. Most threads I see or tools in BST highlight multi-threads as a feature and are proud of crawling 50,000 pages / hour, depending on your internet connection. But what if you...

Hi, I bookmarked the "Distributed Scraping With Multiple Tor Circuits" article from Hacker News. (can't post link). Hope it helps the community. Kind Regards, DT