1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Scraping: Web Based vs Desktop Based

Discussion in 'General Programming Chat' started by macdonjo3, Jan 16, 2014.

  1. macdonjo3

    macdonjo3 Jr. VIP Jr. VIP Premium Member

    Joined:
    Nov 8, 2009
    Messages:
    5,564
    Likes Received:
    4,317
    Location:
    Toronto
    Home Page:
    I was searching Google trying to find some articles or threads arguing web scraping as a desktop app vs scraping as a web app, but my searches didn't return much.

    Where do you guys usually draw the line to decide if your scraper (or app) should be desktop or web based?

    Web based:
    - Easy to run on multiple operating systems
    - Uses the server's resources
    - People pirating your stuff is near impossible since a username/password is required
    - Almost requires a monthly fee so the server costs can be paid
    - Server has to supply the proxies for scraping, or else the server's IP will be banned

    Desktop based:
    - Worry about running on Mac/Windows/Linux and all their versions
    - Uses the users computer and internet resources
    - Pirating is quite easy in most cases
    - Can be sold as a one time fee if the updates required in the future will be minor
    - Users supply proxies

    How do you guys decide if your scraping app would be better as a desktop or web app? Why would you develop on one over the other? I would love to hear some answers.
     
  2. aldragon

    aldragon Power Member

    Joined:
    Aug 5, 2010
    Messages:
    688
    Likes Received:
    192
    Location:
    ^^
    I use both depending on how many client's websites I'm doing at the same time + mine too. I like it way better with a good windows server. VPSs I don't like that much mainly for their low processing power.
    Right now I'm running 2x comps homebased Unusable for regular tasks and 1 big server @ovh . So finally depending on the budget and the amount scraping you want, and how picky you are, I guess both are necessary. To do a Good job you need a lot of scraping and link checking and more.
     
  3. Paper-Boy

    Paper-Boy Elite Member

    Joined:
    Jun 17, 2009
    Messages:
    5,116
    Likes Received:
    1,821
    I always preferred web apps since I don't have to worry about problems not working with my laptop.

    The less problems you give to your potential customer, increases the value of your service/app.

    Just my 2 cents.
     
  4. kideze

    kideze Elite Member

    Joined:
    Jun 23, 2009
    Messages:
    1,719
    Likes Received:
    330
    Location:
    the GRAND valley
    Scrapping? Try Import.io Easy and free.
     
  5. macdonjo3

    macdonjo3 Jr. VIP Jr. VIP Premium Member

    Joined:
    Nov 8, 2009
    Messages:
    5,564
    Likes Received:
    4,317
    Location:
    Toronto
    Home Page:
    Eh, no thanks. Not for me.
     
  6. xenergy81

    xenergy81 Junior Member

    Joined:
    Jul 6, 2009
    Messages:
    105
    Likes Received:
    6
    Occupation:
    Full time Online Marketer & Software Engineer
    Location:
    IPv4
    Assuming you have skills to build both web & desktop apps: I would go with desktop based for my first apps, as you don't need any server maintenance fee. Then move on to web base as I have more money to invest on my own server, etc.
     
  7. macdonjo3

    macdonjo3 Jr. VIP Jr. VIP Premium Member

    Joined:
    Nov 8, 2009
    Messages:
    5,564
    Likes Received:
    4,317
    Location:
    Toronto
    Home Page:
    I have web based servers and I have the money for them. That's not a factor I was looking at.
     
  8. sirgold

    sirgold Supreme Member

    Joined:
    Jun 25, 2010
    Messages:
    1,260
    Likes Received:
    645
    Occupation:
    Busy proving the Pareto principle right
    Location:
    A hot one
    Always on desktop.

    X-platform development these days is trivial and it's way more simple to get a way better performance running native code than overloading a stack of technologies that most of the time execute interpreted code (with a few notable exceptions with bytecode, jit, hhvm and so forth).

    Plus if you're offering a service no point whatsoever in hogging YOUR own servers (that YOU have to pay for...) when you could demand the cpu and network load on your clients' desktops especially if you plan on offering a service. Doesn't really make much sense from a business perspective cost-wise.
     
  9. divok

    divok Senior Member

    Joined:
    Jul 21, 2010
    Messages:
    1,015
    Likes Received:
    634
    Location:
    http://twitter.com/divok
    I would create a free version which would be web based also as a marketing ploy . And sell the advanced one as a desktop version . This would be quite easy only if we code the core in a single language say python or java and create a Gui around it . I am actually planning to do the same .
     
  10. devilishrsps

    devilishrsps Newbie

    Joined:
    Jan 25, 2014
    Messages:
    15
    Likes Received:
    1
    Most of the time I choose desktop, because its much easer to manage and access (for both the programmer and user). Plus, I feel that both executing areas have their own pros. But I prefer desktop because its so easy and great :D
     
  11. macdonjo3

    macdonjo3 Jr. VIP Jr. VIP Premium Member

    Joined:
    Nov 8, 2009
    Messages:
    5,564
    Likes Received:
    4,317
    Location:
    Toronto
    Home Page:
    Ha, why would you have the free version web based? So you'll be paying server resource costs for free customers, but not for paid customers? Weird.
     
  12. divok

    divok Senior Member

    Joined:
    Jul 21, 2010
    Messages:
    1,015
    Likes Received:
    634
    Location:
    http://twitter.com/divok
    suppose its a keyword tracker , I would allow only 10 keywords to be tracked online . And make them come online daily if they want to update (like whatsmyserp.com) and paste a big advertisement about the pro version . Also I could sell related services like link building and proxies . Moreover If i release another product , I already have a huge set of potential buyers .
     
  13. macdonjo3

    macdonjo3 Jr. VIP Jr. VIP Premium Member

    Joined:
    Nov 8, 2009
    Messages:
    5,564
    Likes Received:
    4,317
    Location:
    Toronto
    Home Page:
    You can set limits and place pop ups and ads in desktop software too. This isn't anything exclusive to web based development.



    This isn't related to what we're talking about here, but yes, everyone knows if you sell a product, you can upsell or downsell another product and build a buyer list.
     
  14. s0ap

    s0ap Executive VIP Jr. VIP Premium Member

    Joined:
    Sep 23, 2008
    Messages:
    230
    Likes Received:
    810
    Occupation:
    :] guess
    Location:
    Congo/DRC
    I would code native and deploy it on servers. If this is designed to be sold as a product then I would implement a client-server model in the native code and handle customers with a web UI that it interfaces with. This allows you to sell SaaS and lock people out without having to update the backend, and allows one update to hit everyone at the same time.
     
    • Thanks Thanks x 1
  15. tahworld

    tahworld Regular Member

    Joined:
    Aug 16, 2013
    Messages:
    457
    Likes Received:
    393
    Location:
    ✔✔✔✔✔✔✔
    Hey Mac, I have tried both. They both have their advantages and disadvantages.

    I used PHPCrawl to execute my web based project, and also Scrapy.

    Scrapy is supposed to be faster, but it's really hard to get the right settings.
    PHPCrawl is fast enough for me, especially the multi-threaded option.

    About your proxy concern: PHPCrawl can use a proxy, I'm currently writing a script to make it rotate proxies.

    For computer based Scrapebox of course. Nothing beats Scrapebox.

    Good luck my friend, let me know if you have any questions, it's an interesting topic for sure.
     
  16. MrBlue

    MrBlue Senior Member

    Joined:
    Dec 18, 2009
    Messages:
    950
    Likes Received:
    662
    Occupation:
    Web/Bot Developer
    If you want to protect your code from being pirated and simplify code update deployments, a web based SaaS (Software as a Service) solution is your best route.
     
  17. YouFeelMeDawg?

    YouFeelMeDawg? BANNED BANNED

    Joined:
    Aug 10, 2011
    Messages:
    266
    Likes Received:
    371
    THIS THIS THIS !!!
    Exactly, how am planning on doing a SaaS product once I get enough time and more coding experience
    on the client side(javascript frameworks). For example, say if I was creating a web 2.0 acct creator, I would build the bot native and run an UI on the browser, so the customer can control it from the browser.This can apply the same to social media bots, make the bot run on their vps/server/home pc and have the GUI/UI on the browser.


    off-topic:
    How are you finding c9 for coding. I am actually planning on releasing and open source project that I have been working for a while to bhw. I feel like bhw is missing the open source vibe and every premium project is coded to be sold, and there is a lack of open source projects build for the community.I am not talking about the simple bots that every once in a while a member codes for free(those "I am a programmer, I will make a bot for free" kind of threads) , but actually bots that would be compared to the likes of the paid bots that sell for 57-200'ish that automate one site very well like fb,twitter,tumblr,pinterest etc.
     
  18. TehEpidemick

    TehEpidemick BANNED BANNED

    Joined:
    Oct 31, 2011
    Messages:
    778
    Likes Received:
    181
    right in one. im not sure why you would do a pure web based app.

    Make them use their own internet and CPU, use your servers to check the liscense information and update the bot.
     
  19. kideze

    kideze Elite Member

    Joined:
    Jun 23, 2009
    Messages:
    1,719
    Likes Received:
    330
    Location:
    the GRAND valley
    Use Import.io.

    Easy to use and runs on the web.
     
  20. s0ap

    s0ap Executive VIP Jr. VIP Premium Member

    Joined:
    Sep 23, 2008
    Messages:
    230
    Likes Received:
    810
    Occupation:
    :] guess
    Location:
    Congo/DRC
    Processor time is cheap, mine isn't. I can get the best performance in ANSI C with no windowing and direct access to the socket layer, most users aren't going to know what to do with that.
     
    • Thanks Thanks x 1