1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Custom program search

Discussion in 'General Programming Chat' started by youtalk, Dec 8, 2012.

  1. youtalk

    youtalk Regular Member

    Joined:
    Jul 5, 2012
    Messages:
    337
    Likes Received:
    6
    Occupation:
    Owner
    Location:
    I don't even know anymore
    Ok I've used software like this in the past, but I'm not sure what it would be called, or how to explain it to a developer.

    At my previous company, we had software that would take a part number, and go out to various websites (each site has a different login credentials and layout), log into them, run the search on that part number, and bring back the information to our system. (part number, item name, price, quantity, etc)
    I have no idea what that would be called, but it was great, and saved us so much time.

    How the program worked:
    The part numbers would then sit in basically a bucket, and every 15 minutes that bucket would start running through each one of the sites, and start running the part number searches.
    Once it did the search, and grabbed the data, it would bring it back to our site and insert it into that specific parts record.
    After it ran through all the parts in that bucket, it would wait and start building its next bucket of parts to run through.

    Ok I hope that made sense. Tell me what I'm looking for to have created.
     
    • Thanks Thanks x 1
  2. jazzc

    jazzc Moderator Staff Member Moderator Jr. VIP

    Joined:
    Jan 27, 2009
    Messages:
    2,468
    Likes Received:
    10,152
    It 's called a scraper.

    You tell your developer what information you want to pull and from what sites and he makes it for you. In general the more you bother describing the functionality of what you want in a lot of detail - the better the chances you 'll get what you want fast.
     
  3. youtalk

    youtalk Regular Member

    Joined:
    Jul 5, 2012
    Messages:
    337
    Likes Received:
    6
    Occupation:
    Owner
    Location:
    I don't even know anymore
    Is it a off the shelf program that can be modified for what I need?
     
  4. jazzc

    jazzc Moderator Staff Member Moderator Jr. VIP

    Joined:
    Jan 27, 2009
    Messages:
    2,468
    Likes Received:
    10,152
    No, custom.
     
  5. youtalk

    youtalk Regular Member

    Joined:
    Jul 5, 2012
    Messages:
    337
    Likes Received:
    6
    Occupation:
    Owner
    Location:
    I don't even know anymore
    I'm hearing screen scraping, web fetching, page crawler, web data extraction, and many more. How do I know which one I need or want?
     
  6. supervortex

    supervortex Junior Member

    Joined:
    Aug 8, 2009
    Messages:
    133
    Likes Received:
    10
    Zennoposter can do this task.
     
  7. jazzc

    jazzc Moderator Staff Member Moderator Jr. VIP

    Joined:
    Jan 27, 2009
    Messages:
    2,468
    Likes Received:
    10,152
    All refer to the same thing.
     
  8. youtalk

    youtalk Regular Member

    Joined:
    Jul 5, 2012
    Messages:
    337
    Likes Received:
    6
    Occupation:
    Owner
    Location:
    I don't even know anymore
    Has anyone used scraper wiki?

    I'm not a developer, so I'm not sure how it works.
     
  9. youtalk

    youtalk Regular Member

    Joined:
    Jul 5, 2012
    Messages:
    337
    Likes Received:
    6
    Occupation:
    Owner
    Location:
    I don't even know anymore
    Does anyone know if visual web ripper has a function to log into accounts?

    Or another one like it that's cheaper?