Scraping dynamically loaded Java script websites

S

Saintloki

Junior Member

Nov 17, 2021

104

33

• Jul 7, 2022

• #1

Does anyone know of any good video tutorials or articles on scrapping dynamically loaded Java script websites?
I've searched on YouTube but every single tutorial on scraping websites is on websites where everything is visible in the html. I have not managed to find one that will help me scrape a website whose data is completely loaded through a Java script or hidden api

 

TomTheCat

Elite Member

Oct 27, 2019

2,370

2,749

• Jul 7, 2022

• #2

Code:

https://scrapfly.io/blog/web-scraping-with-selenium-and-python/

 

BlogPro

Elite Member

Jr. Executive VIP

Jr. VIP

Apr 23, 2012

2,575

5,384

• Jul 7, 2022

• #3

There are two ways -

1. First would be where you take the XHR request into consideration, and replace it with an HTTP request. Or stimulate it in your code.

2. Would be using a library with a webdriver / headless browser to stimulate the page load and then run your scrape such as Selenium, Playwright or Pyppeteer etc.

Here's a few I found -

(Using XHR)

 

Smackhead

Regular Member

Dec 22, 2021

389

219

• Jul 7, 2022

• #4

Google puppeteer or selenium the docs and examples should be enough to get you started. Happy to help with any questions you have.

 

irisXE

Regular Member

Jr. VIP

Jun 13, 2022

245

114

• Jul 7, 2022

• #5

Everything @BlogPro and @Smackhead said or just go with a no-code ready made browser automation solution with tons of fingerprints and tricks preloaded in it that will ease your work and save you a LOT of time: Zennoposter (you'll find it here, in the marketplace) or BAS.

 

I

ibobo

Newbie

Oct 9, 2016

38

3

• Jul 11, 2022

• #6

hi do you have the url ? or something similar ?

 

iam_ironman

Power Member

Oct 20, 2014

578

176

• Jul 11, 2022

• #7

share the url, will code in python and PM you