Software like SECockpit and Market Samurai do massive amounts of search engine scraping of Google, Bing etc. A volume of queries that normal proxy services can't handle. Plus - and this is just an assumption - they probably use services that have APIs available for the scraping.
Does anyone have any idea what sort of services these guys use?
This is mostly out of sheer curiosity, and partly because I might be looking to make my own software.
I did a Google search for cloud proxies/cloud scraping etc. and couldn't find much of anything. I did find this site, which claim they can scrape Google at hundreds of queries per minute:
http://www.trustedproxies.com/corporate-private-proxy-server-products/big-g-stealth-extractor.html
But I don't think software like Market Samurai/SECockpit use these guys. I'm really interested in knowing what enterprise level services they use for their SE scraping. If anyone has any clue on this, let me know
Does anyone have any idea what sort of services these guys use?
This is mostly out of sheer curiosity, and partly because I might be looking to make my own software.
I did a Google search for cloud proxies/cloud scraping etc. and couldn't find much of anything. I did find this site, which claim they can scrape Google at hundreds of queries per minute:
http://www.trustedproxies.com/corporate-private-proxy-server-products/big-g-stealth-extractor.html
But I don't think software like Market Samurai/SECockpit use these guys. I'm really interested in knowing what enterprise level services they use for their SE scraping. If anyone has any clue on this, let me know