1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Anyone here advanced in JS?

Discussion in 'General Programming Chat' started by spectrejoe, May 4, 2016.

  1. spectrejoe

    spectrejoe Jr. VIP Jr. VIP

    Joined:
    Sep 25, 2013
    Messages:
    2,105
    Likes Received:
    440
    Home Page:
    I wanted to create a script that would get the names + prices of items from one site and save them locally, then I would go to another site do the same and have a node calculate the price difference between them.

    How hard would this be?
     
  2. Frozen27

    Frozen27 Newbie

    Joined:
    Aug 8, 2013
    Messages:
    20
    Likes Received:
    4
    You need a correlation point (something that correlates the product from site A to equivalent product of site B). I've done this in the past (price tracker), not in JS though. PM me for further info.
     
  3. The Mentalist

    The Mentalist Power Member

    Joined:
    May 8, 2013
    Messages:
    738
    Likes Received:
    269
    Location:
    Inside your head
    This isn't terribly hard. The problem, as anyone who has done lots of web scraping knows, is finding the patterns to consistently and accurately extract data from pages.

    To illustrate, if I am comparing names of a product...

    A simple system will get text inside of this div and compare to the text in the span on another page to make sure the products are right.

    A robust system will shingle and compute the Jaccard similarity of the names, use an algorithm to find likely manufacturing numbers that correlate to the product on the page and compare those between pages. The end result would be a metric/probability that you've identified the same product. Based on a threshold you then decide whether it is or isn't the same product.

    That may seem overkill, but that's often what it takes to get a web scraper to reliably extract data. Because if you make buy/sell decisions from those prices just one bad extraction can cost you big $$$.

    I usually just pray that there is a public API available.
     
  4. MrBlue

    MrBlue Senior Member

    Joined:
    Dec 18, 2009
    Messages:
    974
    Likes Received:
    680
    Occupation:
    Web/Bot Developer
    What are the sites in question? This could be done using either server or client side JS.