Requests vs Selenium
- Thread starter JackTheRooster
- Start date
- Status
PineappleCrush
BANNED
- Jan 20, 2016
- 439
- 52
Oh right then I am in agreement. I actually was under the same impression as you until someone informed me the other day that javascript can be bypassed just by looking at when it sends over the wire after it completes it client side activity. Took me a little to realise this because you need to look not at the root page's requests but rather the slew of requests that comes forth after you complete the js action and the url can be some weird random one which you would emulate.
I did this successfully only on one website so far- not that tried on any others yet- but I was told that you can bypass all js like this.
PS I see most other people here are ignorant as to how requests work and emulate browser activity 1:1 so no need to further the discussion with them. Let them stay in their webdriver bubble thinking they have the upper hand
I do use webdriver sometimes still but it is when I am pressed for time since sometimes it can be tricky to figure out that 1 request which will make the whole thing go and I need an email creator working for instance. If I had the luxury of time I would always at least try and do my bots in requests before moving to webdriver.
JackTheRooster
Registered Member
- Apr 19, 2018
- 50
- 9
Oh right then I am in agreement. I actually was under the same impression as you until someone informed me the other day that javascript can be bypassed just by looking at when it sends over the wire after it completes it client side activity. Took me a little to realise this because you need to look not at the root page's requests but rather the slew of requests that comes forth after you complete the js action and the url can be some weird random one which you would emulate.
I did this successfully only on one website so far- not that tried on any others yet- but I was told that you can bypass all js like this.
PS I see most other people here are ignorant as to how requests work and emulate browser activity 1:1 so no need to further the discussion with them. Let them stay in their webdriver bubble thinking they have the upper hand
I do use webdriver sometimes still but it is when I am pressed for time since sometimes it can be tricky to figure out that 1 request which will make the whole thing go and I need an email creator working for instance. If I had the luxury of time I would always at least try and do my bots in requests before moving to webdriver.
Are you talking about looking at the HTTP post that is issued, or actually executing the JavaScript to force it to make the requests? If it is the latter, do you use some JS library to run the JavaScript? Seems easier than trying to mock the HTTP request via the requests module.
That's quite useful information!
GrandScraperG
BANNED
- Jan 12, 2019
- 52
- 5
Everything could and should be emulates using pure HTTP and reversing the .js
I have been able to scrape the biggest sites and biggest protections out there.
Selenium is shit and take lots of resources and easy to detect as well.
I have been able to scrape the biggest sites and biggest protections out there.
Selenium is shit and take lots of resources and easy to detect as well.
JackTheRooster
Registered Member
- Apr 19, 2018
- 50
- 9
True, would love to do everything pure HTTP, but how do you deal with minified JS? The var names are scrambled and the context is hard to gather.
GrandScraperG
BANNED
- Jan 12, 2019
- 52
- 5
Read a bit about js code obfuscation. you will have to reverse the whole code to understand exactly.
- Status