Are HTTP requests bots worse than browser emulation?

[karupoiss]

I know that HTTP is faster and more scalable but how does it work? For browser you need to fake stuff to not seem as a bot. Wouldn't doing pure HTTP request only not a be a screaming red flag that you are a bot?

 

[BlackDesign]

you are asking good questions , it depends alot on the site automated ,if you simply scrapping information and can do it via request that would be the best,
If you want to do actions on the site/app it depends if they track any other kind of activity (such as mouse movement , clicked location etc...) you should try investigating the requests , if its too complicated you can run a test with request based bot and see how they react

 

[Grudge]

Depends on the amount of logging & detection they have + how much you really care.
At the end of the day, what a browser does can be 100% replicated with http requests.

 

[marcus7777]

Your main problem is not being detected, your problem usually will be interact with browser generated content AKA javascript, thats why you need a real browser.

 

[HydraProxy]

Wouldn't doing pure HTTP request only not a be a screaming red flag that you are a bot?

Some requests library support adding headers, user-agents as well, however you are right.

Using simple requests you do not load any canvas or other browser real-life specific features/triggers.

However, depending on the goal of accessing a website, you might not need full browser interaction and emulation and simple requests will do the job.

For example, scraping for content, data-mining, research tasks - tasks that usually involve retrieving data, rather than manipulating/sending data - can be performed (depending on the website) with requests only.

 

[noellarkin]

Some requests library support adding headers, user-agents as well, however you are right.

Using simple requests you do not load any canvas or other browser real-life specific features/triggers.

However, depending on the goal of accessing a website, you might not need full browser interaction and emulation and simple requests will do the job.

For example, scraping for content, data-mining, research tasks - tasks that usually involve retrieving data, rather than manipulating/sending data - can be performed (depending on the website) with requests only.

Excellent point. I love how HTTP req. have such low overhead and light on the system, but there's a lot of UI manipulation that method can never do. Btw I LOVE the containerization in your HydraHeaders application - - will you be introducing cookie injection anytime soon? That would complete it IMO

 

[TwSylvester]

If site requires for Javascript load, you wont get any informations from HTTP Response, you need to test it out.

 

[satyr85]

Its about amount of time needed to reverse engineer browser requests. Requests in some cases are worse, in some cases are way better. I always try to automate stuff without browser as long as I can.

 

[TheLovelyUnicron]

The only problem is JS, and if you love reverse engineering then you are good to go.

 

[yellowcat]

I know that HTTP is faster and more scalable but how does it work? For browser you need to fake stuff to not seem as a bot. Wouldn't doing pure HTTP request only not a be a screaming red flag that you are a bot?

It all just comes down to pure http requests, all the javascript is doing in terms of bot detection is making http requests back n forth from a browser (which in itself can be stripped apart and automated tho it takes time).

Everything depends on what you are trying to build.
if it's a super l33t bot than yeah start reversing the javascript & making raw http requests.

if its just a quick n dirty script maybe poke around & see if u can make http requests if not just browser automate it.


if the javascript is too much of a pain to reverse and u really want a fast bot you can try to use browser automation to store the cookies then load them via http



Most sites will present different web pages depending on user-agents.
for instance the login page for m.facebook.com has less javascript if ur using an older phone user-agent such as blackberry z30 vs using an iphone 8 user-agent.

check for mobile versions of websites, sometimes they are far less complex in terms of html structure than the none mobile versions "cough m.facebook.com"

automation is just problem solving try to think outside the box & have fun

 

[noellarkin]

It all just comes down to pure http requests, all the javascript is doing in terms of bot detection is making http requests back n forth from a browser (which in itself can be stripped apart and automated tho it takes time).

Everything depends on what you are trying to build.
if it's a super l33t bot than yeah start reversing the javascript & making raw http requests.

if its just a quick n dirty script maybe poke around & see if u can make http requests if not just browser automate it.


if the javascript is too much of a pain to reverse and u really want a fast bot you can try to use browser automation to store the cookies then load them via http



Most sites will present different web pages depending on user-agents.
for instance the login page for m.facebook.com has less javascript if ur using an older phone user-agent such as blackberry z30 vs using an iphone 8 user-agent.

check for mobile versions of websites, sometimes they are far less complex in terms of html structure than the none mobile versions "cough m.facebook.com"

automation is just problem solving try to think outside the box & have fun

lol you just gave away a lot of peoples 'secret methods' right there

 

[yellowcat]

lol you just gave away a lot of peoples 'secret methods' right there

Shh don't tell anyone the secret methods