1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Batch/automatic uncovering "hidden" elements which appear after you click

Discussion in 'General Scripting Chat' started by Pete Friedman, Apr 9, 2015.

  1. Pete Friedman

    Pete Friedman Regular Member

    Joined:
    Apr 6, 2015
    Messages:
    357
    Likes Received:
    159
    Hello :)
    There's a site I want to scrap for some data but the data is not immediately visibile after you load the webpage - to uncover the data you need to click on a button (probably some JavaScript, not a HTML button unfortunately) and then the data appears. There are many such fields on the site.
    What ways can you think of to help me accomplish my task? I'd like to write a short script for that, preferably in Python.
     
  2. potatoes

    potatoes Newbie

    Joined:
    Aug 6, 2011
    Messages:
    32
    Likes Received:
    3
    Use selenium, it handles js
     
  3. MrBlue

    MrBlue Senior Member

    Joined:
    Dec 18, 2009
    Messages:
    974
    Likes Received:
    680
    Occupation:
    Web/Bot Developer
    You can use selenium or headless browsers like phantomjs or casperjs. Both can be controlled using Python.