How to scrape/record Javascript page

E

evertd

Newbie
Joined
Mar 25, 2008
Messages
17
Reaction score
66
Hi,

I am wondering if this is possible, and maybe there is some tool that can already do it"

I want to be able to record/scrape data from a javascript page that is updated every few seconds (without any interaction from my side) and save that data to a database.

Specifically, one section of the page only displays the last 10 actions of other website users and drops off the older actions. I'd like to be able to record all those actions and save them for analysis.

I hope that is clear enough.
Thanks!
 
Advertise on BHW
You 'll probably have to resort to a custom made solution, either with a browser control or via http. Dynamic sites are more complex to parse that static ones, so less tools for that.
 
You have to resort to a custom made solution as jazzc suggested.
Probably the records are returned as JSON, which is very easy to parse. Please give me the details, so i can assist you.
 
theMagicNumber said:
You have to resort to a custom made solution as jazzc suggested.
Probably the records are returned as JSON, which is very easy to parse. Please give me the details, so i can assist you.
Click to expand...

Thanks for the offer. The site is skoreit.com, if you go to skoreit.com/Auctions you see a list of current auctions, if you click on any of them you see the auction details, on that page is a section called Bidding History, and in there it shows the last 10 bids. What I am looking to do is to be able to scrape the whole history of the auction and write it to a database for later analysis.

Thanks!
 
I think I can do that for you. PM me if you want me to handle this task.
 
I also see that the /History/IDGOESHERE page only gives the last 10 records. BUT it is constantly updated during the auction, so you can keep hitting it and get all the data. Thanks MagicNumber!

And thanks for the people that offered to create a tool for this. I think I'm going to try to learn something and attempt to do this for myself.

Thanks for everybody's input!
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Networking Downloads

Marketplace

Account Selling Content / Copywriting Hosting Images, Logos & Videos Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Misc

Making Money

Affiliate Programs Hire a Freelancer Making Money Pay Per Click Site Flipping

BlackHatWorld

BHW Merch Forum Suggestions & Feedback Introductions Lounge Dispute Resolution

Other

Domain Name Forum IM Journeys Web Hosting Newsletter
Back
Top