1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How to scrape/record Javascript page

Discussion in 'General Programming Chat' started by evertd, Jun 5, 2012.

  1. evertd

    evertd Newbie

    Joined:
    Mar 25, 2008
    Messages:
    17
    Likes Received:
    65
    Hi,

    I am wondering if this is possible, and maybe there is some tool that can already do it"

    I want to be able to record/scrape data from a javascript page that is updated every few seconds (without any interaction from my side) and save that data to a database.

    Specifically, one section of the page only displays the last 10 actions of other website users and drops off the older actions. I'd like to be able to record all those actions and save them for analysis.

    I hope that is clear enough.
    Thanks!
     
  2. jazzc

    jazzc Moderator Staff Member Moderator Jr. VIP

    Joined:
    Jan 27, 2009
    Messages:
    2,468
    Likes Received:
    10,143
    You 'll probably have to resort to a custom made solution, either with a browser control or via http. Dynamic sites are more complex to parse that static ones, so less tools for that.
     
  3. theMagicNumber

    theMagicNumber Regular Member

    Joined:
    May 13, 2010
    Messages:
    345
    Likes Received:
    195
    You have to resort to a custom made solution as jazzc suggested.
    Probably the records are returned as JSON, which is very easy to parse. Please give me the details, so i can assist you.
     
  4. evertd

    evertd Newbie

    Joined:
    Mar 25, 2008
    Messages:
    17
    Likes Received:
    65
    Thanks for the offer. The site is skoreit.com, if you go to skoreit.com/Auctions you see a list of current auctions, if you click on any of them you see the auction details, on that page is a section called Bidding History, and in there it shows the last 10 bids. What I am looking to do is to be able to scrape the whole history of the auction and write it to a database for later analysis.

    Thanks!
     
  5. sockpuppet

    sockpuppet Junior Member

    Joined:
    Nov 7, 2011
    Messages:
    155
    Likes Received:
    145
    • Thanks Thanks x 2
    Last edited: Jun 6, 2012
  6. ionutcib

    ionutcib Junior Member

    Joined:
    Feb 10, 2011
    Messages:
    116
    Likes Received:
    8
    Occupation:
    Java Programmer
    I think I can do that for you. PM me if you want me to handle this task.
     
    • Thanks Thanks x 1
  7. facebookcpa

    facebookcpa BANNED BANNED

    Joined:
    Mar 3, 2012
    Messages:
    364
    Likes Received:
    98
    Will do this tool for you if you want, and even cheap. PM
     
    • Thanks Thanks x 1
  8. theMagicNumber

    theMagicNumber Regular Member

    Joined:
    May 13, 2010
    Messages:
    345
    Likes Received:
    195
    • Thanks Thanks x 2
  9. sockpuppet

    sockpuppet Junior Member

    Joined:
    Nov 7, 2011
    Messages:
    155
    Likes Received:
    145
    but that only returns 10 results or am i missing someting?
     
  10. evertd

    evertd Newbie

    Joined:
    Mar 25, 2008
    Messages:
    17
    Likes Received:
    65
    I also see that the /History/IDGOESHERE page only gives the last 10 records. BUT it is constantly updated during the auction, so you can keep hitting it and get all the data. Thanks MagicNumber!

    And thanks for the people that offered to create a tool for this. I think I'm going to try to learn something and attempt to do this for myself.

    Thanks for everybody's input!
     
  11. sockpuppet

    sockpuppet Junior Member

    Joined:
    Nov 7, 2011
    Messages:
    155
    Likes Received:
    145
    lol, i was missing the obvious solution