Anyone here advanced in JS?

I wanted to create a script that would get the names + prices of items from one site and save them locally, then I would go to another site do the same and have a node calculate the price difference between them.

How hard would this be?

You need a correlation point (something that correlates the product from site A to equivalent product of site B). I've done this in the past (price tracker), not in JS though. PM me for further info.

This isn't terribly hard. The problem, as anyone who has done lots of web scraping knows, is finding the patterns to consistently and accurately extract data from pages.

To illustrate, if I am comparing names of a product...

A simple system will get text inside of this div and compare to the text in the span on another page to make sure the products are right.

A robust system will shingle and compute the Jaccard similarity of the names, use an algorithm to find likely manufacturing numbers that correlate to the product on the page and compare those between pages. The end result would be a metric/probability that you've identified the same product. Based on a threshold you then decide whether it is or isn't the same product.

That may seem overkill, but that's often what it takes to get a web scraper to reliably extract data. Because if you make buy/sell decisions from those prices just one bad extraction can cost you big $$$.

I usually just pray that there is a public API available.

spectrejoe said:

I wanted to create a script that would get the names + prices of items from one site and save them locally, then I would go to another site do the same and have a node calculate the price difference between them.

Click to expand...

What are the sites in question? This could be done using either server or client side JS.

The Mentalist said:

I usually just pray that there is a public API available.

Click to expand...

Newb question. I apologize in advance. How can you tell if a site has a public API? I have been manually listing a wholesaler's catalogue...its slowly dimming my soul

tryagain2day said:

Newb question. I apologize in advance. How can you tell if a site has a public API? I have been manually listing a wholesaler's catalogue...its slowly dimming my soul

Click to expand...

Well it depends on your skill level with web development. Not sure if this makes sense to you, but you can right click on a webpage, then select "Inspect" or "Inspect Element". Once you've done that you should see your browser's developer tools. You want to go to the "Network" section, and check for any XHR or AJAX calls. See where those calls go, and you may find an API endpoint for some of their data. If you don't mind sharing you could just post the URL here.

The Mentalist said:

XHR or AJAX calls. See where those calls go, and you may find an API endpoint for some of their data.

Click to expand...

XHR is blank and AJAX column is not even present

That means the site doesn't have a public API, or at least the page you're viewing isn't using one. Because it's not making any AJAX calls, the results are directly rendered into HTML.