Are HTTP requests bots worse than browser emulation?
- Thread starter karupoiss
- Start date
BlackDesign
Power Member
- Sep 8, 2014
- 581
- 265
you are asking good questions , it depends alot on the site automated ,if you simply scrapping information and can do it via request that would be the best,
If you want to do actions on the site/app it depends if they track any other kind of activity (such as mouse movement , clicked location etc...) you should try investigating the requests , if its too complicated you can run a test with request based bot and see how they react
If you want to do actions on the site/app it depends if they track any other kind of activity (such as mouse movement , clicked location etc...) you should try investigating the requests , if its too complicated you can run a test with request based bot and see how they react
marcus7777
Regular Member
- Apr 30, 2011
- 348
- 306
Your main problem is not being detected, your problem usually will be interact with browser generated content AKA javascript, thats why you need a real browser.
- Jun 6, 2020
- 1,000
- 310
Some requests library support adding headers, user-agents as well, however you are right.
Using simple requests you do not load any canvas or other browser real-life specific features/triggers.
However, depending on the goal of accessing a website, you might not need full browser interaction and emulation and simple requests will do the job.
For example, scraping for content, data-mining, research tasks - tasks that usually involve retrieving data, rather than manipulating/sending data - can be performed (depending on the website) with requests only.
noellarkin
Senior Member
- Mar 14, 2021
- 1,006
- 1,492
Excellent point. I love how HTTP req. have such low overhead and light on the system, but there's a lot of UI manipulation that method can never do. Btw I LOVE the containerization in your HydraHeaders application - - will you be introducing cookie injection anytime soon? That would complete it IMO
- Apr 23, 2011
- 192
- 32
If site requires for Javascript load, you wont get any informations from HTTP Response, you need to test it out.
TheLovelyUnicron
Newbie
- Sep 9, 2021
- 15
- 5
The only problem is JS, and if you love reverse engineering then you are good to go.
yellowcat
Regular Member
- Aug 27, 2015
- 404
- 297
It all just comes down to pure http requests, all the javascript is doing in terms of bot detection is making http requests back n forth from a browser (which in itself can be stripped apart and automated tho it takes time).
Everything depends on what you are trying to build.
if it's a super l33t bot than yeah start reversing the javascript & making raw http requests.
if its just a quick n dirty script maybe poke around & see if u can make http requests if not just browser automate it.
if the javascript is too much of a pain to reverse and u really want a fast bot you can try to use browser automation to store the cookies then load them via http
Most sites will present different web pages depending on user-agents.
for instance the login page for m.facebook.com has less javascript if ur using an older phone user-agent such as blackberry z30 vs using an iphone 8 user-agent.
check for mobile versions of websites, sometimes they are far less complex in terms of html structure than the none mobile versions
"cough m.facebook.com"automation is just problem solving try to think outside the box & have fun
noellarkin
Senior Member
- Mar 14, 2021
- 1,006
- 1,492
lol you just gave away a lot of peoples 'secret methods' right thereIt all just comes down to pure http requests, all the javascript is doing in terms of bot detection is making http requests back n forth from a browser (which in itself can be stripped apart and automated tho it takes time).
Everything depends on what you are trying to build.
if it's a super l33t bot than yeah start reversing the javascript & making raw http requests.
if its just a quick n dirty script maybe poke around & see if u can make http requests if not just browser automate it.
if the javascript is too much of a pain to reverse and u really want a fast bot you can try to use browser automation to store the cookies then load them via http
Most sites will present different web pages depending on user-agents.
for instance the login page for m.facebook.com has less javascript if ur using an older phone user-agent such as blackberry z30 vs using an iphone 8 user-agent.
check for mobile versions of websites, sometimes they are far less complex in terms of html structure than the none mobile versions
automation is just problem solving try to think outside the box & have fun
yellowcat
Regular Member
- Aug 27, 2015
- 404
- 297
Shh don't tell anyone the secret methodslol you just gave away a lot of peoples 'secret methods' right there