what to scrape?

Discussion in 'General Programming Chat' started by blazed, Sep 5, 2008.

  1. blazed

    blazed Junior Member

    Joined:
    Aug 15, 2008
    Messages:
    178
    Likes Received:
    119
    So I've been playing with some code after a long long hiatus from programming and I want to try my hand at writing a spider with the intention of building a database. I see the most retarded databases selling like hot cakes on DP - databases that are nothing more but scraped content from sites like wikipedia, ezinearticles, IMDB, etc... I'm trying to think of a site to crawl & scrape that would at least leave me with something useful in the end, but I'm drawing blanks. Anyone have any ideas? If I end up building something you can bet I'll throw the DB into member downloads :) Help me think of something!
     
  2. headspin

    headspin Regular Member

    Joined:
    Jun 3, 2008
    Messages:
    235
    Likes Received:
    143
    How about a database of articles sorted by niche? That should sell pretty well. And there are a bunch of open-source libraries that can help you determine which category to fit an article in.