I've already created a distributed scraping system using google cloud, aws, azure & digitalocean to run my remote agents that receive commands to do things (get a url, parse it for blah patterns, return matches, etc..) so I can easily scale up & down from 1 bot to 1000+. The capabilities are insane! I've started to populate my datawarehouse with app store's like google play & webstore, itunes, etc.. with all apps, their ranking , all comments, etc and I hit them once a day so I can then run predictive analysis o determine what is the upcoming trends for apps that people are willing to pay for -- bam, idea list with solid profit potential
I've definitely created this platform & framework to be able to handle any type of scraping+storing+performing analytics over massive data stores using hadoop & mapreduce.
So implementing this really isn't that far fetched!
What would you say something like this is worth? (Given that it would be cheap -- no upfront licensing crap, just pay for usage with a free starter package)
I'd like to get some more background information so ping me on skype @appsoa or post there.