1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

what to scrape?

Discussion in 'General Programming Chat' started by blazed, Sep 5, 2008.

  1. blazed

    blazed Junior Member

    Joined:
    Aug 15, 2008
    Messages:
    178
    Likes Received:
    119
    So I've been playing with some code after a long long hiatus from programming and I want to try my hand at writing a spider with the intention of building a database. I see the most retarded databases selling like hot cakes on DP - databases that are nothing more but scraped content from sites like wikipedia, ezinearticles, IMDB, etc... I'm trying to think of a site to crawl & scrape that would at least leave me with something useful in the end, but I'm drawing blanks. Anyone have any ideas? If I end up building something you can bet I'll throw the DB into member downloads :) Help me think of something!
     
  2. headspin

    headspin Regular Member

    Joined:
    Jun 3, 2008
    Messages:
    234
    Likes Received:
    140
    Home Page:
    How about a database of articles sorted by niche? That should sell pretty well. And there are a bunch of open-source libraries that can help you determine which category to fit an article in.