Scraping: Web Based vs Desktop Based

[macdonjo3]

I was searching Google trying to find some articles or threads arguing web scraping as a desktop app vs scraping as a web app, but my searches didn't return much.

Where do you guys usually draw the line to decide if your scraper (or app) should be desktop or web based?

Web based:
- Easy to run on multiple operating systems
- Uses the server's resources
- People pirating your stuff is near impossible since a username/password is required
- Almost requires a monthly fee so the server costs can be paid
- Server has to supply the proxies for scraping, or else the server's IP will be banned

Desktop based:
- Worry about running on Mac/Windows/Linux and all their versions
- Uses the users computer and internet resources
- Pirating is quite easy in most cases
- Can be sold as a one time fee if the updates required in the future will be minor
- Users supply proxies

How do you guys decide if your scraping app would be better as a desktop or web app? Why would you develop on one over the other? I would love to hear some answers.

 

[aldragon]

I use both depending on how many client's websites I'm doing at the same time + mine too. I like it way better with a good windows server. VPSs I don't like that much mainly for their low processing power.
Right now I'm running 2x comps homebased Unusable for regular tasks and 1 big server @ovh . So finally depending on the budget and the amount scraping you want, and how picky you are, I guess both are necessary. To do a Good job you need a lot of scraping and link checking and more.

 

[Paper-Boy]

I always preferred web apps since I don't have to worry about problems not working with my laptop.

The less problems you give to your potential customer, increases the value of your service/app.

Just my 2 cents.

 

[kideze]

Scrapping? Try Import.io Easy and free.

 

[macdonjo3]

Scrapping? Try Import.io Easy and free.

Eh, no thanks. Not for me.

 

[xenergy81]

Assuming you have skills to build both web & desktop apps: I would go with desktop based for my first apps, as you don't need any server maintenance fee. Then move on to web base as I have more money to invest on my own server, etc.

 

[macdonjo3]

Assuming you have skills to build both web & desktop apps: I would go with desktop based for my first apps, as you don't need any server maintenance fee. Then move on to web base as I have more money to invest on my own server, etc.

I have web based servers and I have the money for them. That's not a factor I was looking at.

 

[sirgold]

Always on desktop.

X-platform development these days is trivial and it's way more simple to get a way better performance running native code than overloading a stack of technologies that most of the time execute interpreted code (with a few notable exceptions with bytecode, jit, hhvm and so forth).

Plus if you're offering a service no point whatsoever in hogging YOUR own servers (that YOU have to pay for...) when you could demand the cpu and network load on your clients' desktops especially if you plan on offering a service. Doesn't really make much sense from a business perspective cost-wise.

 

[divok]

I would create a free version which would be web based also as a marketing ploy . And sell the advanced one as a desktop version . This would be quite easy only if we code the core in a single language say python or java and create a Gui around it . I am actually planning to do the same .

 

[devilishrsps]

Most of the time I choose desktop, because its much easer to manage and access (for both the programmer and user). Plus, I feel that both executing areas have their own pros. But I prefer desktop because its so easy and great

 

[macdonjo3]

I would create a free version which would be web based also as a marketing ploy . And sell the advanced one as a desktop version . This would be quite easy only if we code the core in a single language say python or java and create a Gui around it . I am actually planning to do the same .

Ha, why would you have the free version web based? So you'll be paying server resource costs for free customers, but not for paid customers? Weird.

 

[divok]

Ha, why would you have the free version web based? So you'll be paying server resource costs for free customers, but not for paid customers? Weird.

suppose its a keyword tracker , I would allow only 10 keywords to be tracked online . And make them come online daily if they want to update (like whatsmyserp.com) and paste a big advertisement about the pro version . Also I could sell related services like link building and proxies . Moreover If i release another product , I already have a huge set of potential buyers .

 

[macdonjo3]

suppose its a keyword tracker , I would allow only 10 keywords to be tracked online . And make them come online daily if they want to update (like whatsmyserp.com) and paste a big advertisement about the pro version .

You can set limits and place pop ups and ads in desktop software too. This isn't anything exclusive to web based development.

Also I could sell related services like link building and proxies . Moreover If i release another product , I already have a huge set of potential buyers .

This isn't related to what we're talking about here, but yes, everyone knows if you sell a product, you can upsell or downsell another product and build a buyer list.

 

[s0ap]

I would code native and deploy it on servers. If this is designed to be sold as a product then I would implement a client-server model in the native code and handle customers with a web UI that it interfaces with. This allows you to sell SaaS and lock people out without having to update the backend, and allows one update to hit everyone at the same time.

 

[tahworld]

Hey Mac, I have tried both. They both have their advantages and disadvantages.

I used PHPCrawl to execute my web based project, and also Scrapy.

Scrapy is supposed to be faster, but it's really hard to get the right settings.
PHPCrawl is fast enough for me, especially the multi-threaded option.

About your proxy concern: PHPCrawl can use a proxy, I'm currently writing a script to make it rotate proxies.

For computer based Scrapebox of course. Nothing beats Scrapebox.

Good luck my friend, let me know if you have any questions, it's an interesting topic for sure.

 

[MrBlue]

If you want to protect your code from being pirated and simplify code update deployments, a web based SaaS (Software as a Service) solution is your best route.

 

[YouFeelMeDawg?]

I would code native and deploy it on servers. If this is designed to be sold as a product then I would implement a client-server model in the native code and handle customers with a web UI that it interfaces with. This allows you to sell SaaS and lock people out without having to update the backend, and allows one update to hit everyone at the same time.

THIS THIS THIS !!!
Exactly, how am planning on doing a SaaS product once I get enough time and more coding experience
on the client side(javascript frameworks). For example, say if I was creating a web 2.0 acct creator, I would build the bot native and run an UI on the browser, so the customer can control it from the browser.This can apply the same to social media bots, make the bot run on their vps/server/home pc and have the GUI/UI on the browser.


off-topic:
How are you finding c9 for coding. I am actually planning on releasing and open source project that I have been working for a while to bhw. I feel like bhw is missing the open source vibe and every premium project is coded to be sold, and there is a lack of open source projects build for the community.I am not talking about the simple bots that every once in a while a member codes for free(those "I am a programmer, I will make a bot for free" kind of threads) , but actually bots that would be compared to the likes of the paid bots that sell for 57-200'ish that automate one site very well like fb,twitter,tumblr,pinterest etc.

 

[TehEpidemick]

I would code native and deploy it on servers. If this is designed to be sold as a product then I would implement a client-server model in the native code and handle customers with a web UI that it interfaces with. This allows you to sell SaaS and lock people out without having to update the backend, and allows one update to hit everyone at the same time.

right in one. im not sure why you would do a pure web based app.

Make them use their own internet and CPU, use your servers to check the liscense information and update the bot.

 

[kideze]

Use Import.io.

Easy to use and runs on the web.

 

[s0ap]

Processor time is cheap, mine isn't. I can get the best performance in ANSI C with no windowing and direct access to the socket layer, most users aren't going to know what to do with that.