Hi, I am wondering if this is possible, and maybe there is some tool that can already do it" I want to be able to record/scrape data from a javascript page that is updated every few seconds (without any interaction from my side) and save that data to a database. Specifically, one section of the page only displays the last 10 actions of other website users and drops off the older actions. I'd like to be able to record all those actions and save them for analysis. I hope that is clear enough. Thanks!
You 'll probably have to resort to a custom made solution, either with a browser control or via http. Dynamic sites are more complex to parse that static ones, so less tools for that.
You have to resort to a custom made solution as jazzc suggested. Probably the records are returned as JSON, which is very easy to parse. Please give me the details, so i can assist you.
Thanks for the offer. The site is skoreit.com, if you go to skoreit.com/Auctions you see a list of current auctions, if you click on any of them you see the auction details, on that page is a section called Bidding History, and in there it shows the last 10 bids. What I am looking to do is to be able to scrape the whole history of the auction and write it to a database for later analysis. Thanks!
It is comet/bayeux using this lib http://www.frozenmountain.com/websync/ I would say the best way is to use some client implementation of Comet/Bayeux like "CometD Java client".
Hello, it is easier than you think. The bid information is returned as JSON. You have to call https://www.skoreit.com/Auctions/History/IDGOESHERE to get the records. For example: https://www.skoreit.com/Auctions/History/269681 I can code you a simple C#/VB.NET solution to scrape the database and extract the history. I won't charge you anything, however i am going to work in my spare time so it might take a few days.
I also see that the /History/IDGOESHERE page only gives the last 10 records. BUT it is constantly updated during the auction, so you can keep hitting it and get all the data. Thanks MagicNumber! And thanks for the people that offered to create a tool for this. I think I'm going to try to learn something and attempt to do this for myself. Thanks for everybody's input!
