Scraping JS-heavy + iframe pages

i'm trying to scrape a web application that's built using AngularJS, that has a javascript generated token that needs to be validated by the backend and several iframe blocks on the page and it's also using strange data formats, probably a variation of protobuff, the data looks all garbled coming back from the API.

Automated logins are blocked by this JavaScript token, so it's very hard to scrape data via the REST calls, the page has to render in a browser before you can login or see any data in a usable format, the REST calls also garble up the data so that you're not looking at easy parsable JSON or similar.
The iFrame further makes it non-selenium friendly

Does a tool exist that can scrape and interact with pages that were made very scraper-unfriendly (JS-login token, garbled REST data and iframes everywhere)?
Currently building a tool that screenshots the page and uses OCR to find positions of text / images in order to find coordinates on the screen where a click should happen, but this feels overkill for what i'm trying to do.

Take a look at headless browser, like Selenium

Sheepp said:

Take a look at headless browser, like Selenium

Click to expand...

Selenium uses a JavaScript driver to interact with the browser, it can't do that to a protected Iframe

If you are looking for screenshots, puppeteer might help. It uses headless chrome under the hood.

Just curious, have you tried mimicking the request headers using curl? What are you getting? garbled text?

rft120 said:

Just curious, have you tried mimicking the request headers using curl? What are you getting? garbled text?

Click to expand...

Site uses a strange frontend-generated token to make sure that the requests are sent from a browser and not from curl / any other H-T-T-P (apparently h-t-t-p is seen as a link by the forum, facepalm) post utility.
And that token changes with every request, each request uses a different token and if i re-use the same token on a subsequent request, i get logged out.
Even if i could get around the token issue, the data looks like it's compressed, not gzip compressed, but rather in a format that JavaScript can decompress again.
The JavaScript itself is also minified, so it's very hard to see what's going on in its millions of lines of garbled JavaScript.

Thanks for the suggestion, will check out the tool and see if it gets around the random iframes
Some of the buttons / forms are in iframes just to make it more difficult for tools like selenium to get in there.

Is it distil-network protected site?

Maybe could build one with NightmareJS or PhantomJS

If you will ocr it then could use AutoIT old version to help parts of it yeah thats overkill and messy but easier yo.

Re@PeR said:

Thanks for the suggestion, will check out the tool and see if it gets around the random iframes
Some of the buttons / forms are in iframes just to make it more difficult for tools like selenium to get in there.

Click to expand...

use puppeteer, it uses the chrome-devtools-protocol to control the browser
so you can do everything that you can do in the chrome devtools panel
frames: https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-frame