How to scrape/record Javascript page

evertd

Newbie
Joined
Mar 25, 2008
Messages
17
Reaction score
66
Hi,

I am wondering if this is possible, and maybe there is some tool that can already do it"

I want to be able to record/scrape data from a javascript page that is updated every few seconds (without any interaction from my side) and save that data to a database.

Specifically, one section of the page only displays the last 10 actions of other website users and drops off the older actions. I'd like to be able to record all those actions and save them for analysis.

I hope that is clear enough.
Thanks!
 
You 'll probably have to resort to a custom made solution, either with a browser control or via http. Dynamic sites are more complex to parse that static ones, so less tools for that.
 
You have to resort to a custom made solution as jazzc suggested.
Probably the records are returned as JSON, which is very easy to parse. Please give me the details, so i can assist you.
 
You have to resort to a custom made solution as jazzc suggested.
Probably the records are returned as JSON, which is very easy to parse. Please give me the details, so i can assist you.

Thanks for the offer. The site is skoreit.com, if you go to skoreit.com/Auctions you see a list of current auctions, if you click on any of them you see the auction details, on that page is a section called Bidding History, and in there it shows the last 10 bids. What I am looking to do is to be able to scrape the whole history of the auction and write it to a database for later analysis.

Thanks!
 
I think I can do that for you. PM me if you want me to handle this task.
 
I also see that the /History/IDGOESHERE page only gives the last 10 records. BUT it is constantly updated during the auction, so you can keep hitting it and get all the data. Thanks MagicNumber!

And thanks for the people that offered to create a tool for this. I think I'm going to try to learn something and attempt to do this for myself.

Thanks for everybody's input!
 
Back
Top