1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

what to scrape?

Discussion in 'General Programming Chat' started by blazed, Sep 5, 2008.

  1. blazed

    blazed Junior Member

    Aug 15, 2008
    Likes Received:
    So I've been playing with some code after a long long hiatus from programming and I want to try my hand at writing a spider with the intention of building a database. I see the most retarded databases selling like hot cakes on DP - databases that are nothing more but scraped content from sites like wikipedia, ezinearticles, IMDB, etc... I'm trying to think of a site to crawl & scrape that would at least leave me with something useful in the end, but I'm drawing blanks. Anyone have any ideas? If I end up building something you can bet I'll throw the DB into member downloads :) Help me think of something!
  2. headspin

    headspin Regular Member

    Jun 3, 2008
    Likes Received:
    Home Page:
    How about a database of articles sorted by niche? That should sell pretty well. And there are a bunch of open-source libraries that can help you determine which category to fit an article in.