Sending https requests on python

Soo i finished automate the boring stuff with python course and i started writing few bots just to sharpen my skills and pickup valuable knowledge im trying to scrape some data off websites and i thought why not the site i learned on (udemy.com) im trying to login to the website but no luck i tried using selenium and it detects it easily so i used the requests module however i didnot find the right url to post the login request to and right parameters?? i thought i identify which kind of authentication it uses and build up from there i tried sending a unautherised request in order to get a 401 error and just read the www-authenticate variable in the header but no luck either but viewing the source code i think its a bearer-type authentication??? It sends a csrf something token along with the username and pass how do i obtain that?? I can use bs4 to implement it in my code as every token is newly generated with every account and session?? Anyway i v been strugling for the past week any kind of advice will be HUGELY APPRECIATED and thank you in advance

I suggest you install POSTMAN, it's Chrome extension, it allows you to capture packets that get sent or recieved by/to the server you're targeting. Usually the form requests are made with a POST request, when you capture the network source, you'll see how the formdata is filled, then you go from there by sending the server a formdata with the required infos. I am not on my pc right now, but I'll take a look when I do and give you a more appropriate answer ^^

Are you on windows? Download FIDDLER and use that to monitor the requests from your browser then replicate.

If on Mac, then I think CHARLES is comparable, although I have no experience with that

rootjazz said:

Are you on windows? Download FIDDLER and use that to monitor the requests from your browser then replicate.

If on Mac, then I think CHARLES is comparable, although I have no experience with that

Click to expand...

Fiddler was tbh useless i just used the dev tools in chrome and it was much simpler and easier to understand i already made a successful login i felt so proud lol im stuck with recaptcha now

The login system probably checks default headers for browsers.

An apache module will refuse requests from python scripts if the headers don't match the real browser. That's what that 401 looks like to me.

bartosimpsonio said:

The login system probably checks default headers for browsers.

An apache module will refuse requests from python scripts if the headers don't match the real browser. That's what that 401 looks like to me.

Click to expand...

Yeap also the referer