Are HTTP requests bots worse than browser emulation?

karupoiss

Elite Member
Joined
May 16, 2012
Messages
2,559
Reaction score
926
I know that HTTP is faster and more scalable but how does it work? For browser you need to fake stuff to not seem as a bot. Wouldn't doing pure HTTP request only not a be a screaming red flag that you are a bot?
 

BlackDesign

Regular Member
Joined
Sep 8, 2014
Messages
496
Reaction score
159
you are asking good questions , it depends alot on the site automated ,if you simply scrapping information and can do it via request that would be the best,
If you want to do actions on the site/app it depends if they track any other kind of activity (such as mouse movement , clicked location etc...) you should try investigating the requests , if its too complicated you can run a test with request based bot and see how they react :)
 

Grudge

Jr. VIP
Jr. VIP
Joined
Nov 2, 2018
Messages
345
Reaction score
337
Website
grudge-services.com
Depends on the amount of logging & detection they have + how much you really care.
At the end of the day, what a browser does can be 100% replicated with http requests.
 

marcus7777

Jr. VIP
Jr. VIP
Joined
Apr 30, 2011
Messages
308
Reaction score
242
Your main problem is not being detected, your problem usually will be interact with browser generated content AKA javascript, thats why you need a real browser.
 

HydraProxy

Jr. VIP
Jr. VIP
Joined
Jun 6, 2020
Messages
510
Reaction score
231
Website
hydraproxy.com
Wouldn't doing pure HTTP request only not a be a screaming red flag that you are a bot?

Some requests library support adding headers, user-agents as well, however you are right.

Using simple requests you do not load any canvas or other browser real-life specific features/triggers.

However, depending on the goal of accessing a website, you might not need full browser interaction and emulation and simple requests will do the job.

For example, scraping for content, data-mining, research tasks - tasks that usually involve retrieving data, rather than manipulating/sending data - can be performed (depending on the website) with requests only.
 

noellarkin

Regular Member
Joined
Mar 14, 2021
Messages
204
Reaction score
243
Some requests library support adding headers, user-agents as well, however you are right.

Using simple requests you do not load any canvas or other browser real-life specific features/triggers.

However, depending on the goal of accessing a website, you might not need full browser interaction and emulation and simple requests will do the job.

For example, scraping for content, data-mining, research tasks - tasks that usually involve retrieving data, rather than manipulating/sending data - can be performed (depending on the website) with requests only.
Excellent point. I love how HTTP req. have such low overhead and light on the system, but there's a lot of UI manipulation that method can never do. Btw I LOVE the containerization in your HydraHeaders application - - will you be introducing cookie injection anytime soon? That would complete it IMO
 

araXis

Jr. VIP
Jr. VIP
Joined
Apr 23, 2011
Messages
102
Reaction score
16
Website
instantadsense.com
If site requires for Javascript load, you wont get any informations from HTTP Response, you need to test it out.
 

satyr85

Jr. VIP
Jr. VIP
Joined
Aug 7, 2011
Messages
1,189
Reaction score
1,064
Its about amount of time needed to reverse engineer browser requests. Requests in some cases are worse, in some cases are way better. I always try to automate stuff without browser as long as I can.
 
Joined
Sep 9, 2021
Messages
15
Reaction score
5
The only problem is JS, and if you love reverse engineering then you are good to go.
 

yellowcat

Regular Member
Joined
Aug 27, 2015
Messages
363
Reaction score
249
I know that HTTP is faster and more scalable but how does it work? For browser you need to fake stuff to not seem as a bot. Wouldn't doing pure HTTP request only not a be a screaming red flag that you are a bot?
It all just comes down to pure http requests, all the javascript is doing in terms of bot detection is making http requests back n forth from a browser (which in itself can be stripped apart and automated tho it takes time).

Everything depends on what you are trying to build.
if it's a super l33t bot than yeah start reversing the javascript & making raw http requests.

if its just a quick n dirty script maybe poke around & see if u can make http requests if not just browser automate it.


if the javascript is too much of a pain to reverse and u really want a fast bot you can try to use browser automation to store the cookies then load them via http :)



Most sites will present different web pages depending on user-agents.
for instance the login page for m.facebook.com has less javascript if ur using an older phone user-agent such as blackberry z30 vs using an iphone 8 user-agent.

check for mobile versions of websites, sometimes they are far less complex in terms of html structure than the none mobile versions :) "cough m.facebook.com"

automation is just problem solving try to think outside the box & have fun
 

noellarkin

Regular Member
Joined
Mar 14, 2021
Messages
204
Reaction score
243
It all just comes down to pure http requests, all the javascript is doing in terms of bot detection is making http requests back n forth from a browser (which in itself can be stripped apart and automated tho it takes time).

Everything depends on what you are trying to build.
if it's a super l33t bot than yeah start reversing the javascript & making raw http requests.

if its just a quick n dirty script maybe poke around & see if u can make http requests if not just browser automate it.


if the javascript is too much of a pain to reverse and u really want a fast bot you can try to use browser automation to store the cookies then load them via http :)



Most sites will present different web pages depending on user-agents.
for instance the login page for m.facebook.com has less javascript if ur using an older phone user-agent such as blackberry z30 vs using an iphone 8 user-agent.

check for mobile versions of websites, sometimes they are far less complex in terms of html structure than the none mobile versions :) "cough m.facebook.com"

automation is just problem solving try to think outside the box & have fun
lol you just gave away a lot of peoples 'secret methods' right there :)
 
Top