After weeks of brainstorming/trialling I finally have a list of every single listing from a major directory that has approx 4 million listings. Now my problem is this, I need to grab the external URLs from each page but there just aren't enough proxies in the world it seems. I'm using Scrapebox's link extractor and my private proxies are good for a few hundred scrapes but then get banned. Scrapebox's own proxies are mostly dead by the time you've checked them and started using them so I'm not getting much success from them. I'm stumped, can anyone offer any other methods?