1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Scraping still profitable?

Discussion in 'Cloaking and Content Generators' started by huzah, Apr 19, 2012.

  1. huzah

    huzah Newbie

    Joined:
    Nov 10, 2010
    Messages:
    41
    Likes Received:
    10
    I've been programming as a hobby (but mostly to learn) for a few years now, but considering the fact that I am almost 22 and soon on my own, the times of ladi-daïng(©) are over. It's time to start making some money, and fast.

    I built a scraping backend using a combo of C#, PHP, mysql and javascript that supports proxies (don't want to be blacklisted by any sites). It's pretty sophisticated; scalable, proper error handling, easy to add extra scraping jobs etc etc.

    Example of the scraper:
    Code:
    http://bit.ly/HUipmz
    Note that you must click the button twice in that example, once for the search to start, and again to get the results (if they're even there by the second try).

    My question to you is; is there any money to be made in this at all? Because duplicate content is just that.. duplicate. I think that if I monetize it with @ds3ns3, I would risk losing my @ds3ns3 account. Although I'm not sure... as long as the website is actually good looking, functional, and unique overall (when you add all the elements together), maybe I won't get my butt banned.

    My first idea with this is to create an index of ALL pc games in the history of ever & ever. Once I scraped the names of the games, all I got to do to generate content is to utilize my scraper software to automate searches to, as an example, get related youtube videos, a game description from some website like gamespot/IGN/whatever, get pictures from google images, maybe an RSS feed or two and so on and so forth. And in case there's still any money to be made in linking to warez maybe also add searches to kat.ph/thepiratebay/mininova etc. Although then adsense is definitely a no-no.

    Once that's done I could also write some bots to automatically promote the content, but I'm too new to this to know exactly where & what.

    Any thoughts on all of this, anyone?
     
  2. NTG98

    NTG98 Junior Member

    Joined:
    May 27, 2009
    Messages:
    128
    Likes Received:
    12
    Occupation:
    Student
    Location:
    Texas
    As long as there is data, scraping is profitable if you can get creative with it. The power is in the ideas. For example, look at SpyFu. They built a huge business upon screen scraping google results.

    You can PM me if you want to talk about scraping, because I'm currently undertaking quite a few scraping projects.
     
  3. llamedo

    llamedo Junior Member

    Joined:
    Mar 11, 2011
    Messages:
    135
    Likes Received:
    26
    Location:
    Atlanta
    That tool looks great, how do you integrate the proxies? Are you using any special hosting? Or just a shared one?
     
  4. huzah

    huzah Newbie

    Joined:
    Nov 10, 2010
    Messages:
    41
    Likes Received:
    10
    PM'd ;D

    Well the website runs on a dedicated server, but that's not really important. The proxy is *selected* from a mysql database, based on how often it has been used (I allow a maximum usage of 25 pages/proxy/hour/website but also no more than 4 pages/proxy/minute/website). It is then sent along with the scraping request to the C# Master server, which then selects a child server for the actual scraping. It is inside that C# Child server that the proxy is actually used. They're just squid proxies running on cheap VPS's.

    By the way, it's funny how many of the test searches are sex related, lmao. (Don't worry, no IP's are being logged).
     
  5. iwillnbd

    iwillnbd Regular Member

    Joined:
    Mar 13, 2012
    Messages:
    398
    Likes Received:
    363
    Occupation:
    ....
    Location:
    C:
    what kind of program do you plan on writing for promoting the content !?
     
  6. mcbaine

    mcbaine Newbie

    Joined:
    Apr 24, 2012
    Messages:
    16
    Likes Received:
    0
    sex is the only thing on the internet isnt it?
     
  7. huzah

    huzah Newbie

    Joined:
    Nov 10, 2010
    Messages:
    41
    Likes Received:
    10
    Not sure yet. I don't want to re-invent the wheel so I'd probably use some already existing tools such as scrapebox. And then áfter I find something that works, I can start thinking about automating that process.

    Judging by the searches committed by you guys, it's sex, food, drugs and cars: http://bit.ly/JvPNBf

    lol ;D
     
  8. WebDev

    WebDev Regular Member

    Joined:
    Oct 31, 2010
    Messages:
    384
    Likes Received:
    484
    Gender:
    Male
    Location:
    UK
    You're probably a couple of years too late - would have been good commercially as a wordpress autoblog plugin

    I'd look more to "data mining" - e.g. finding stuff for researchers e.g. case law for lawyers, medical research for doctors, etc. not as something for webmasters