Scraping still profitable?

Discussion in 'Cloaking and Content Generators' started by huzah, Apr 19, 2012.

  1. huzah

    huzah Newbie

    Joined:
    Nov 10, 2010
    Messages:
    41
    Likes Received:
    10
     
  2. NTG98

    NTG98 Junior Member

    Joined:
    May 27, 2009
    Messages:
    128
    Likes Received:
    12
    Occupation:
    Student
    Location:
    Texas
    As long as there is data, scraping is profitable if you can get creative with it. The power is in the ideas. For example, look at SpyFu. They built a huge business upon screen scraping google results.

    You can PM me if you want to talk about scraping, because I'm currently undertaking quite a few scraping projects.
     
  3. llamedo

    llamedo Junior Member

    Joined:
    Mar 11, 2011
    Messages:
    135
    Likes Received:
    26
    Location:
    Atlanta
    That tool looks great, how do you integrate the proxies? Are you using any special hosting? Or just a shared one?
     
  4. huzah

    huzah Newbie

    Joined:
    Nov 10, 2010
    Messages:
    41
    Likes Received:
    10
    PM'd ;D

    Well the website runs on a dedicated server, but that's not really important. The proxy is *selected* from a mysql database, based on how often it has been used (I allow a maximum usage of 25 pages/proxy/hour/website but also no more than 4 pages/proxy/minute/website). It is then sent along with the scraping request to the C# Master server, which then selects a child server for the actual scraping. It is inside that C# Child server that the proxy is actually used. They're just squid proxies running on cheap VPS's.

    By the way, it's funny how many of the test searches are sex related, lmao. (Don't worry, no IP's are being logged).
     
  5. iwillnbd

    iwillnbd Power Member

    Joined:
    Mar 13, 2012
    Messages:
    503
    Likes Received:
    415
    Occupation:
    ....
    Location:
    C:
    what kind of program do you plan on writing for promoting the content !?
     
  6. mcbaine

    mcbaine Newbie

    Joined:
    Apr 24, 2012
    Messages:
    16
    Likes Received:
    0
    sex is the only thing on the internet isnt it?
     
  7. huzah

    huzah Newbie

    Joined:
    Nov 10, 2010
    Messages:
    41
    Likes Received:
    10
     
  8. WebDev

    WebDev Regular Member

    Joined:
    Oct 31, 2010
    Messages:
    384
    Likes Received:
    501
    Gender:
    Male
    Location:
    UK
    You're probably a couple of years too late - would have been good commercially as a wordpress autoblog plugin

    I'd look more to "data mining" - e.g. finding stuff for researchers e.g. case law for lawyers, medical research for doctors, etc. not as something for webmasters