A good URL Harvester/Scraper

retry

Junior Member
Joined
Oct 1, 2008
Messages
102
Reaction score
48
I'm in need of a good URL Harvester/Scraper. It dosn't matter if it's free or not.

It will need proxy support.

Can you suggest something?

Regards

Retry
 
A really good one comes with xrumer, hrefer. You can easily get custom 200k urls a day on a normal broadband home connection. Than pr sort them with another tool in hrefer.
 
I'm also in need of a good scraper.

I need one that can spider redirect links at chamber of commerce and business association sites. Most large metro coc sites seem to use obfuscators or redirects to send visitors to coc member urls, instead of just listing the actual urls. Xenu link sleuth works ok but is bottoming out when it comes to following some obfuscated 'next' buttons. Won't traverse to the next pages on some sites.

So, do you know of a harvester that can go deep and give me a list of all the urls listed in a coc's database? I don't need pagerank or any other seo feature [grabbing whois info along the ways could be helpful but not required].

thx
 
ScrapeBox is a great scraper. Well worth the money, take a look at Sweetfunny's sig link.
 
I use
Code:
www.clextractor.com
, can download for free. Extract, bulk mail for free, up to 15 emails.
 
I've got a free one around here somewhere, gimme a little bit and I'll PM you a link
 
I did buy ScrapeBox for a couple of weeks ago, and I'm very happy with the product:)
 
Hrefer for SERP scraping, Visual Web Spider for custom sites scraping.
 
http://gscrape.com/

It's online, JS based and 100% client side so your keywords are secure, it doesn't get banned (ever) and you can scrape unlimited urls.

Hasn't got all the bells and whistles some other paid pieces of software have, but it's quick, it works and it's free.
 
Anyone having issue from harvesting url from google and yahoo recently?
 
Anyone having issue from harvesting url from google and yahoo recently?

Nope - going fine for me.

What sort of problems are you having - I'll help if I can.
 
Anyone here know a good harvester for blogs/forums? I want a software that does it both

G00B3R

It depends on what footprints you use when searching. So most if not all, if they can do one, they can do the other.

As stated above an excellent one is Hrefer that comes with Xrumer. Second to Hrefer i would say Scrapebox which for it's price does a kickass job.
 
@SpamHat: Nice code! Thanks to Firefox to allow post to cross domain through ajax.
 
Back
Top