Looking for a tutorial on HTTP Requests

Hi there!

I’m currently using python and selenium to automate my stuff, but I’d like to try out http requests as I’ve heard it’s easier + takes less resources

Does anybody have a recommendation?

It's true that you'll use less resources and maybe in some cases it could be easier but.. you will need understand cookies and headers to authentice.

Python docs for httpclient should be enough to get your started.
https://docs.python.org/3/library/http.client.html
I'd recommended making the request yourself in your browser with the network tab open, right click the request and then copy to something you can read probably curl. This will give you all of the components sent by the browser as part of request. Most of the headers will not be required but you'll need to locate things like bearer tokens/ sessionids, useragent is usually important.

Play around sending the request without some of parts to figure out what's needed.

You'll learn alot from messing around with it.

I wil try to help if you have any questions just drop me a message.

Enjoy

Wussabi said:

Hi there!

I’m currently using python and selenium to automate my stuff, but I’d like to try out http requests as I’ve heard it’s easier + takes less resources

Does anybody have a recommendation?

Click to expand...

Everyone slags off vb6, cause it's so old, but it still works, I build bots in a couple of hrs using this. No doubt the critics will say "what vb6" , but u know what, it still works.

https://www.blackhatworld.com/seo/tutorial-scrape-apis-web-pages-data-using-c.1305321/
That link includes HTTP programming course but in c#

I dont really know of a tutorial, but you can begin to learn how requests work by opening your browsers console (ctrl+shift+i) and looking at the network tab. You can load a website and look at the headers, cookies, and responses the website sent back.

That being said web scraping using pure http requests vs selenium or some other browser automater is way more work. The modern web is cluttered with pointless javascript. So make sure it makes sense before you commit a lot of time to doing it.

To inspect, edit, re-send, create new HTTP requests, get Tamper Dev extension if you're using Chrome. You can also get Burp Suite to get a deeper insight. I've made many bots inspecting the requests with Tamper Dev.

You can use Postman to make a request and it can generate a code instantly for any language. reqbin.com also can do this online. You can use chrome devtools to find out requests and parameters for different projects.

Whoever says that it is easy to work with queries is gently telling a lie. You will have to emulate the work of the browser, execute each request manually and dig into the javascript code. Selenium is simpler but requires more resources. We use requests where it is justified, usually web scraping.

Chrome browser is good for checking what requests are being performed. Just open Dev Tools (right click - Inspect), then Network tab. Good luck with hacking!

There are several Python libraries that can be used for making HTTP requests, including Requests, httplib, and urllib. Requests is the most user-friendly, while httplib and urllib offer more control for advanced use cases.