Beyond HTTP Requests

noellarkin · Aug 6, 2021

Automation via HTTP requests and JSON decoding is working out for me for some older sites. No such luck for newer ones, though. I've been using Browser automation for the more popular sites (Reddit etc).

Of course, there's a night-and-day difference in terms of the time taken. I'm wondering if there's any way HTTP request methods of automation can be extended and applied to websites that are heavy in JS (which is, at the moment, the major stumbling block for me).

Right now, I know how to do HTTP GET and POST, I know JSON decoding, RegExp. I'm really new to programming (compared to most of you here) so I'm wondering if there are any knowledge gaps that are preventing me from being able to work with JS heavy sites. Is there something I ought to learn, some advanced HTTP concept, or just accept that browser automation is the only way forward?

mrwootyeet · Aug 6, 2021

For browser automation, you may be able to use headless selenium + microservices (like GCP cloud functions or AWS lambda functions).

It takes a bit of troubleshooting (at least with python), not sure about the JS side.

I've seen members on here talking about using puppeteer (a headless node.JS API).

Azure Spark · Aug 7, 2021

You don't need browser automation for JS-heavy sites. In the network tab of your browser's devtools, inspect what API calls are made when you take an action on the page, (click the "XHR" filter). Then simply perform a request with the same parameters (and maybe headers) in your scraper.

AlbertoMR3 · Aug 11, 2021

noellarkin said:
Automation via HTTP requests and JSON decoding is working out for me for some older sites. No such luck for newer ones, though. I've been using Browser automation for the more popular sites (Reddit etc).

Of course, there's a night-and-day difference in terms of the time taken. I'm wondering if there's any way HTTP request methods of automation can be extended and applied to websites that are heavy in JS (which is, at the moment, the major stumbling block for me).

Right now, I know how to do HTTP GET and POST, I know JSON decoding, RegExp. I'm really new to programming (compared to most of you here) so I'm wondering if there are any knowledge gaps that are preventing me from being able to work with JS heavy sites. Is there something I ought to learn, some advanced HTTP concept, or just accept that browser automation is the only way forward?

Ofcourse you can... what you need is to check on Chrome console the requests and understand what they are passing. Even if the site is pute JS, the only way to communicate with the backend is via GET,POST,PUT,DELETE and you can always see what the front is sending to o the back end and what the backend responses.

Your difficult with some sites is that they may be using bearer tokens, cookies or other dynamic key generated on the front. That key can be dinamically generated by JS or by serverside on load.
If you see that all the requests share the same bearer, token or cookie, you just need to mimic thisose headers on your request and you will sucesfully been able yo to comunicate via CURL from your script with the site backend

noellarkin · Aug 14, 2021

AlbertoMR3 said:
Ofcourse you can... what you need is to check on Chrome console the requests and understand what they are passing. Even if the site is pute JS, the only way to communicate with the backend is via GET,POST,PUT,DELETE and you can always see what the front is sending to o the back end and what the backend responses.

Your difficult with some sites is that they may be using bearer tokens, cookies or other dynamic key generated on the front. That key can be dinamically generated by JS or by serverside on load.
If you see that all the requests share the same bearer, token or cookie, you just need to mimic thisose headers on your request and you will sucesfully been able yo to comunicate via CURL from your script with the site backend

Thanks for this, it helps me understand the situation a little better. I've been trying to do this for a particular site, where upon login, just like you said, a dynamic session code is generated by the site. I can't seem to figure out what the logic of this code is - - I tried using username+password and base64 encoding it etc etc, just guessing at this point.
Is there a cheatsheet I can use as to the common methods by which authentication tokens, cookies and session codes are generated?

Beyond HTTP Requests

noellarkin

Senior Member

mrwootyeet

Newbie

Azure Spark

Newbie

AlbertoMR3

Junior Member

noellarkin

Senior Member

Main Menu

Marketplace

Making Money

BlackHat World