A good URL Harvester/Scraper


Junior Member
Oct 1, 2008
Reaction score
I'm in need of a good URL Harvester/Scraper. It dosn't matter if it's free or not.

It will need proxy support.

Can you suggest something?


A really good one comes with xrumer, hrefer. You can easily get custom 200k urls a day on a normal broadband home connection. Than pr sort them with another tool in hrefer.
ScrapeBox will do this with public/private proxies and check Pagerank (shameless plug) plus Neta1o has one that's fairly decent a totally free one and a paid one. Pretty sure his free one will take one proxy, plus will scrape content and not just URL's.
I'm also in need of a good scraper.

I need one that can spider redirect links at chamber of commerce and business association sites. Most large metro coc sites seem to use obfuscators or redirects to send visitors to coc member urls, instead of just listing the actual urls. Xenu link sleuth works ok but is bottoming out when it comes to following some obfuscated 'next' buttons. Won't traverse to the next pages on some sites.

So, do you know of a harvester that can go deep and give me a list of all the urls listed in a coc's database? I don't need pagerank or any other seo feature [grabbing whois info along the ways could be helpful but not required].

ScrapeBox is a great scraper. Well worth the money, take a look at Sweetfunny's sig link.
I use
, can download for free. Extract, bulk mail for free, up to 15 emails.
I've got a free one around here somewhere, gimme a little bit and I'll PM you a link
I did buy ScrapeBox for a couple of weeks ago, and I'm very happy with the product:)
Hrefer for SERP scraping, Visual Web Spider for custom sites scraping.

It's online, JS based and 100% client side so your keywords are secure, it doesn't get banned (ever) and you can scrape unlimited urls.

Hasn't got all the bells and whistles some other paid pieces of software have, but it's quick, it works and it's free.
Anyone having issue from harvesting url from google and yahoo recently?
Anyone having issue from harvesting url from google and yahoo recently?

Nope - going fine for me.

What sort of problems are you having - I'll help if I can.
Anyone here know a good harvester for blogs/forums? I want a software that does it both


It depends on what footprints you use when searching. So most if not all, if they can do one, they can do the other.

As stated above an excellent one is Hrefer that comes with Xrumer. Second to Hrefer i would say Scrapebox which for it's price does a kickass job.
@SpamHat: Nice code! Thanks to Firefox to allow post to cross domain through ajax.
AdBlock Detected

We get it, advertisements are annoying!

Sure, ad-blocking software does a great job at blocking ads, but it also blocks useful features and essential functions on BlackHatWorld and other forums. These functions are unrelated to ads, such as internal links and images. For the best site experience please disable your AdBlocker.

I've Disabled AdBlock