I'm in need of a good URL Harvester/Scraper. It dosn't matter if it's free or not. It will need proxy support. Can you suggest something? Regards Retry
A really good one comes with xrumer, hrefer. You can easily get custom 200k urls a day on a normal broadband home connection. Than pr sort them with another tool in hrefer.
ScrapeBox will do this with public/private proxies and check Pagerank (shameless plug) plus Neta1o has one that's fairly decent a totally free one and a paid one. Pretty sure his free one will take one proxy, plus will scrape content and not just URL's.
I'm also in need of a good scraper. I need one that can spider redirect links at chamber of commerce and business association sites. Most large metro coc sites seem to use obfuscators or redirects to send visitors to coc member urls, instead of just listing the actual urls. Xenu link sleuth works ok but is bottoming out when it comes to following some obfuscated 'next' buttons. Won't traverse to the next pages on some sites. So, do you know of a harvester that can go deep and give me a list of all the urls listed in a coc's database? I don't need pagerank or any other seo feature [grabbing whois info along the ways could be helpful but not required]. thx
I use Code: www.clextractor.com , can download for free. Extract, bulk mail for free, up to 15 emails.
http://gscrape.com/ It's online, JS based and 100% client side so your keywords are secure, it doesn't get banned (ever) and you can scrape unlimited urls. Hasn't got all the bells and whistles some other paid pieces of software have, but it's quick, it works and it's free.
It depends on what footprints you use when searching. So most if not all, if they can do one, they can do the other. As stated above an excellent one is Hrefer that comes with Xrumer. Second to Hrefer i would say Scrapebox which for it's price does a kickass job.
