Hiring A Software Developer

[MatthewWoodward]

Hi,

I want to get a custom application developed which will essentially take a list of URL's and cycle through them looking for strings of text and then output a table based report looking up the data it retrieved against a database to return values.

Hope that makes sense without giving away my full idea

I would want the software to have a very nice looking & easy to use user interface and is must be multi threaded able to cope with upto 50 threads.

The problem is I have never tackled anything like this before and would like some advice on what I should be asking potential developers before making a decision.

I imagine it is tricky to find someone that can produce both efficient / lightweight code and a visually pleasing interface?

I also have no idea how much a project like this should cost - I'm planning to give this away to people free of charge so cost is going to be important.

What advice can you offer to someone like me stepping into getting software developed on their behalf for the first time?

 

[WizGizmo]

Thread moved to the "Hire-A-Freelancer" section.

"Wiz"

 

[MatthewWoodward]

Hi Wiz,

I dont want to actually hire anyone I was more looking for advice from experienced developers on what to look for - for example one person will swear by one language over another even though it is not neccessarily the best choice and only want they are comfortable with.

Getting lots of PM's from people assuming I'm looking to hire someone immediately which I dont

 

[NitroCan]

OP, the thread titleis misleading which therefore causing confusion

 

[BladeMcCool]

There are many ways to approach this. You could go for a custom desktop (windows or macos) application, or you could have a linux server based solution with a backend job and a web based frontend, or other ways. I build scraper/poster bots like this with perl that run multiple processes up to whatever memory allows and can do front end admin web gui interfaces that run on linux vps's, so naturally thats the method i would suggest. Myself I can tell you that I'm great at logic and coding and functionality but making stuff look 'visually pleasing' is definitely not my strong point. You may find similar with other developers with strong and weak points. But at any rate, there are standard things out there like twitter bootstrap that any good developer should be able to adapt to whatever templating system they are using for a web interface I like MySQL for a database myself.

For the code that scrapes websites and extracts strings and maps to another list for a lookup value or something, if its that straight forward it shouldnt take more than a handful of hours to develop for someone with multiprocess scraper/poster bot frameworks handy. And then another handful of hours to set it up on your own server and develop the web interface. Maybe even less for the person with the perfect toolkit for getting working and pretty things deployed quickly I typically work for USD $35 per hour (due to global wage arbitrage he he).

 

[MatthewWoodward]

OP, the thread titleis misleading which therefore causing confusion

Ahhh yes I see how that is misleading now =D

There are many ways to approach this. You could go for a custom desktop (windows or macos) application, or you could have a linux server based solution with a backend job and a web based frontend, or other ways. I build scraper/poster bots like this with perl that run multiple processes up to whatever memory allows and can do front end admin web gui interfaces that run on linux vps's, so naturally thats the method i would suggest. Myself I can tell you that I'm great at logic and coding and functionality but making stuff look 'visually pleasing' is definitely not my strong point. You may find similar with other developers with strong and weak points. But at any rate, there are standard things out there like twitter bootstrap that any good developer should be able to adapt to whatever templating system they are using for a web interface I like MySQL for a database myself.

For the code that scrapes websites and extracts strings and maps to another list for a lookup value or something, if its that straight forward it shouldnt take more than a handful of hours to develop for someone with multiprocess scraper/poster bot frameworks handy. And then another handful of hours to set it up on your own server and develop the web interface. Maybe even less for the person with the perfect toolkit for getting working and pretty things deployed quickly I typically work for USD $35 per hour (due to global wage arbitrage he he).

I would definitely want it to be a native windows desktop application which I'm guessing rules you out

The interface design just needs to be simple, clear and uncluttered, no ridicolous colours and large buttons everywhere - you know the type I mean

So a rough estimation based on someone having a scraper framework would be 5-8 hours end to end? Doesn't seem to bad! Ideally I would like to keep it under the $250 mark if possible

 

[jazzc]

There are two types of scrapers - those that harvest static content and those that harvest dynamic content (i.e. content generated by JavaScript and not directly embedded in the code).

The static scrapers need low resources (cpu, memory). The dynamic scrapers all use some kind of browser implementation and thus are memory and cpu hungry.

With a static scraper, 50 threads is more than ok. With a dynamic scraper, not really without a monster computer.

Scraping and graphical interface have nothing to do with each other - two completely different things.

 

[BlueTurtle]

There are two types of scrapers - those that harvest static content and those that harvest dynamic content (i.e. content generated by JavaScript and not directly embedded in the code).

The static scrapers need low resources (cpu, memory). The dynamic scrapers all use some kind of browser implementation and thus are memory and cpu hungry.

With a static scraper, 50 threads is more than ok. With a dynamic scraper, not really without a monster computer.

Scraping and graphical interface have nothing to do with each other - two completely different things.

It depends on the javascript. 99% of it can be parsed with code and doesn't need full browser automation.

Browser automation is only needed for convenience or for some rare cases where parsing javascript isn't feasible like for the google adwords keywords tool. It's basically java that generates obfuscated javascript to decode the keyword data. For that there's no other way than browser automation, but for 99% of scrapers you can do it all manually without a browser.

If javascript is 'generating' the content then all you have to do is watch the HTTP packets to see the request that the javascript has the browser make, then make that request yourself inside your code. The only reason the keyword tool is problematic is because you need to be able to decode the data which is done in a different way every time you login or refresh.

Also, you can easily have 50 browser instances on average hardware. It only becomes a problem when you're talking about 100's approaching the 1000's of procs/threads/instances/connections.

 

[jazzc]

Browser automation is only needed for convenience or for some rare cases where parsing javascript isn't feasible like for the google adwords keywords tool. It's basically java that generates obfuscated javascript to decode the keyword data. For that there's no other way than browser automation, but for 99% of scrapers you can do it all manually without a browser.

Certainly you can study the raw traffic and simulate the business logic found in the javascript code. But depending on the target 's implementation and defenses, that can become tedious fast.

Also, you can easily have 50 browser instances on average hardware. It only becomes a problem when you're talking about 100's approaching the 1000's of procs/threads/instances/connections.

For a very high profile web site, my full stack action emulator crashes with 10 threads on a test VM with 1GB RAM and a single cpu. And although lots of RAM is not a problem, lots of cpu is. Having 50 browsers working at the same time will be very slow, as the cpu will not be able to accommodate them. So, yes, it will work, but with poor performance. Of course, if speed is not a requirement, then sure, why not.