Scraping still profitable?
- Thread starter huzah
- Start date
As long as there is data, scraping is profitable if you can get creative with it. The power is in the ideas. For example, look at SpyFu. They built a huge business upon screen scraping google results.
You can PM me if you want to talk about scraping, because I'm currently undertaking quite a few scraping projects.
You can PM me if you want to talk about scraping, because I'm currently undertaking quite a few scraping projects.
PM'd ;D
Well the website runs on a dedicated server, but that's not really important. The proxy is *selected* from a mysql database, based on how often it has been used (I allow a maximum usage of 25 pages/proxy/hour/website but also no more than 4 pages/proxy/minute/website). It is then sent along with the scraping request to the C# Master server, which then selects a child server for the actual scraping. It is inside that C# Child server that the proxy is actually used. They're just squid proxies running on cheap VPS's.
By the way, it's funny how many of the test searches are sex related, lmao. (Don't worry, no IP's are being logged).
WebDev
Regular Member
- Oct 31, 2010
- 384
- 503
You're probably a couple of years too late - would have been good commercially as a wordpress autoblog plugin
I'd look more to "data mining" - e.g. finding stuff for researchers e.g. case law for lawyers, medical research for doctors, etc. not as something for webmasters