1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Hiring A Software Developer

Discussion in 'Hire a Freelancer' started by MatthewWoodward, Sep 27, 2012.

  1. MatthewWoodward

    MatthewWoodward Jr. VIP Jr. VIP Premium Member

    Joined:
    Aug 31, 2012
    Messages:
    992
    Likes Received:
    1,680
    Occupation:
    SEO
    Location:
    UK
    Home Page:
    Hi,

    I want to get a custom application developed which will essentially take a list of URL's and cycle through them looking for strings of text and then output a table based report looking up the data it retrieved against a database to return values.

    Hope that makes sense without giving away my full idea :p

    I would want the software to have a very nice looking & easy to use user interface and is must be multi threaded able to cope with upto 50 threads.

    The problem is I have never tackled anything like this before and would like some advice on what I should be asking potential developers before making a decision.

    I imagine it is tricky to find someone that can produce both efficient / lightweight code and a visually pleasing interface?

    I also have no idea how much a project like this should cost - I'm planning to give this away to people free of charge so cost is going to be important.

    What advice can you offer to someone like me stepping into getting software developed on their behalf for the first time?
     
  2. WizGizmo

    WizGizmo Super Moderator Staff Member Premium Member

    Joined:
    Mar 28, 2008
    Messages:
    3,833
    Likes Received:
    55,431
    Thread moved to the "Hire-A-Freelancer" section.

    "Wiz"
     
  3. MatthewWoodward

    MatthewWoodward Jr. VIP Jr. VIP Premium Member

    Joined:
    Aug 31, 2012
    Messages:
    992
    Likes Received:
    1,680
    Occupation:
    SEO
    Location:
    UK
    Home Page:
    Hi Wiz,

    I dont want to actually hire anyone I was more looking for advice from experienced developers on what to look for - for example one person will swear by one language over another even though it is not neccessarily the best choice and only want they are comfortable with.

    Getting lots of PM's from people assuming I'm looking to hire someone immediately which I dont :p
     
  4. NitroCan

    NitroCan Jr. VIP Jr. VIP Premium Member

    Joined:
    May 14, 2009
    Messages:
    1,060
    Likes Received:
    259
    Home Page:
    OP, the thread titleis misleading which therefore causing confusion:)
     
    Last edited: Sep 27, 2012
  5. BladeMcCool

    BladeMcCool Newbie

    Joined:
    Oct 24, 2010
    Messages:
    7
    Likes Received:
    8
    Occupation:
    perl/mysql/apache/linux (and a whole lot more)
    Location:
    Victoria BC
    Home Page:
    There are many ways to approach this. You could go for a custom desktop (windows or macos) application, or you could have a linux server based solution with a backend job and a web based frontend, or other ways. I build scraper/poster bots like this with perl that run multiple processes up to whatever memory allows and can do front end admin web gui interfaces that run on linux vps's, so naturally thats the method i would suggest. Myself I can tell you that I'm great at logic and coding and functionality but making stuff look 'visually pleasing' is definitely not my strong point. You may find similar with other developers with strong and weak points. But at any rate, there are standard things out there like twitter bootstrap that any good developer should be able to adapt to whatever templating system they are using for a web interface :) I like MySQL for a database myself.

    For the code that scrapes websites and extracts strings and maps to another list for a lookup value or something, if its that straight forward it shouldnt take more than a handful of hours to develop for someone with multiprocess scraper/poster bot frameworks handy. And then another handful of hours to set it up on your own server and develop the web interface. Maybe even less for the person with the perfect toolkit for getting working and pretty things deployed quickly :) I typically work for USD $35 per hour (due to global wage arbitrage he he).
     
  6. MatthewWoodward

    MatthewWoodward Jr. VIP Jr. VIP Premium Member

    Joined:
    Aug 31, 2012
    Messages:
    992
    Likes Received:
    1,680
    Occupation:
    SEO
    Location:
    UK
    Home Page:
    Ahhh yes I see how that is misleading now =D

    I would definitely want it to be a native windows desktop application which I'm guessing rules you out :)

    The interface design just needs to be simple, clear and uncluttered, no ridicolous colours and large buttons everywhere - you know the type I mean :p

    So a rough estimation based on someone having a scraper framework would be 5-8 hours end to end? Doesn't seem to bad! Ideally I would like to keep it under the $250 mark if possible
     
  7. jazzc

    jazzc Moderator Staff Member Moderator Jr. VIP

    Joined:
    Jan 27, 2009
    Messages:
    2,468
    Likes Received:
    10,143
    There are two types of scrapers - those that harvest static content and those that harvest dynamic content (i.e. content generated by JavaScript and not directly embedded in the code).

    The static scrapers need low resources (cpu, memory). The dynamic scrapers all use some kind of browser implementation and thus are memory and cpu hungry.

    With a static scraper, 50 threads is more than ok. With a dynamic scraper, not really without a monster computer.

    Scraping and graphical interface have nothing to do with each other - two completely different things.
     
    • Thanks Thanks x 1
  8. BlueTurtle

    BlueTurtle BANNED BANNED

    Joined:
    Nov 30, 2011
    Messages:
    700
    Likes Received:
    1,307

    It depends on the javascript. 99% of it can be parsed with code and doesn't need full browser automation.

    Browser automation is only needed for convenience or for some rare cases where parsing javascript isn't feasible like for the google adwords keywords tool. It's basically java that generates obfuscated javascript to decode the keyword data. For that there's no other way than browser automation, but for 99% of scrapers you can do it all manually without a browser.

    If javascript is 'generating' the content then all you have to do is watch the HTTP packets to see the request that the javascript has the browser make, then make that request yourself inside your code. The only reason the keyword tool is problematic is because you need to be able to decode the data which is done in a different way every time you login or refresh.

    Also, you can easily have 50 browser instances on average hardware. It only becomes a problem when you're talking about 100's approaching the 1000's of procs/threads/instances/connections.
     
    Last edited: Sep 28, 2012
  9. jazzc

    jazzc Moderator Staff Member Moderator Jr. VIP

    Joined:
    Jan 27, 2009
    Messages:
    2,468
    Likes Received:
    10,143
    Certainly you can study the raw traffic and simulate the business logic found in the javascript code. But depending on the target 's implementation and defenses, that can become tedious fast.

    For a very high profile web site, my full stack action emulator crashes with 10 threads on a test VM with 1GB RAM and a single cpu. And although lots of RAM is not a problem, lots of cpu is. Having 50 browsers working at the same time will be very slow, as the cpu will not be able to accommodate them. So, yes, it will work, but with poor performance. Of course, if speed is not a requirement, then sure, why not.