1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

What Data Would You Scrape?

Discussion in 'BlackHat Lounge' started by simplegnome, Aug 10, 2012.

Tags:
  1. simplegnome

    simplegnome Newbie

    Joined:
    May 1, 2012
    Messages:
    0
    Likes Received:
    0
    Hey you guys,

    I am starting to think about writing a web app that scrapes data and while I have a few ideas, I really want to create something that will be of use to many rather than just a few.

    In the past I have mainly build private price checkers, twitter intelligence tools and keyword scrapers but I want to push myself to build something that everyone can use (fo free).

    The current ideas are -

    * Clickbank product analytics tool - Pulls the daily feed from CB and analyses rising products based on gravity change and ranking change
    * Amazon product tool - Similar to the CB idea. Picks rising product trends and carries out the keyword research to spot easy wins.

    As this is something I want to build to release for free, I was wondering if anyone had any ideas for scraper and more importantly a source(s) to scrape?
     
  2. utuxia

    utuxia BANNED BANNED

    Joined:
    Feb 14, 2011
    Messages:
    673
    Likes Received:
    111
    there are already a ton of keyword scrapers on the market. i personally want to see an alternative to kontentmachine.com -- i was going to think about writing something like this myself, but I'm a web developer not a desktop developer.
     
  3. simplegnome

    simplegnome Newbie

    Joined:
    May 1, 2012
    Messages:
    0
    Likes Received:
    0
    I like the thinking. It would be interesting to look at linguistic patterns used on forums/blogs etc and find genuine spintax ideas based on the findings.
     
  4. utuxia

    utuxia BANNED BANNED

    Joined:
    Feb 14, 2011
    Messages:
    673
    Likes Received:
    111
    i don't really know what they do, if they just integrate w/ a content spinner or not. That's all I would want. Something quick and easy to make my seo task creation a bit easier. Linguistics patterns in forums? That doesn't sound like something quick and easy.
     
  5. simplegnome

    simplegnome Newbie

    Joined:
    May 1, 2012
    Messages:
    0
    Likes Received:
    0
    Yeah maybe I am over thinking this $hit - the patterns would evolve over time but would depend on having a massive datastore of content to draw upon.