The Best URL Harvester

Scrapebox does well and there is always Hrefer. But when your trying to harvest every blog and forum on the planet is a scraper the best option? Or is there a spider which runs off and just keeps going and going?

How about web crawlers for instance?

der.brain said:

Good Question is there some kind of scraper which can store in a MySql DB
written in PHP for 24 hour scraping?

Click to expand...

i can build one for you
i have an imdb scraper

I have read a lot about crawlers or internet bots. There are a lot of projects already there and even open source.

They are used for indexing for search.

With a big enough db you can slowly crawl the web. The reason for using google and other search engines is to empower the scraping tool filtered by a search term.

The problem is that even if you use huge keyword lists to alter the search results in google, you still end up with like 90% dupes. Trust me I know this all too well.

I have tried everything I can think of, any suggestions?

I did a thing where I loaded the alexa top 1,000,000 urls into a table, then used curl to index the content of them, then ran my fingerprint check / form identification algorithm to copy them into the proper tables if there was a match. I don't have it anymore but I could do it again, and with extra urls like forum.domain.com, domain.com/forum, etc.