Requests vs Selenium

Status
Not open for further replies.
Advertise on BHW
adverts said:
Headless raw requests are easily detected. Unless you want to write your own EMCAScript parser, I'd recommend using Selenium. This assumes you want to emulate "organic" traffic.
Click to expand...
Yes, I'm def going for a mixed approach. Using requests where I can, Selenium where I must.
 
adverts said:
Headless raw requests are easily detected. Unless you want to write your own EMCAScript parser, I'd recommend using Selenium. This assumes you want to emulate "organic" traffic.
Click to expand...

Selenium as a whole is easily detected most of the time. But I heard that you can use MultiLogin with Selenium, and control their own browsers with it.
 
夢市片 said:
Selenium as a whole is easily detected most of the time. But I heard that you can use MultiLogin with Selenium, and control their own browsers with it.
Click to expand...

Are you referring to selenium headless/framework or chromedriver/firefox driver? The latter 2 are actually very difficult to detect with traffic rewriting and injection since you're in full control of the browser stack.
 
adverts said:
Are you referring to selenium headless/framework or chromedriver/firefox driver? The latter 2 are actually very difficult to detect with traffic rewriting and injection since you're in full control of the browser stack.
Click to expand...

Well, not really experienced in that, but I think that it goes together. From what I read on StackOverflow, there was some issue with some website with Selenium + Chrome Webdriver in headless mode, and the site was being able to detect there was some automation going on.
 
夢市片 said:
Well, not really experienced in that, but I think that it goes together. From what I read on StackOverflow, there was some issue with some website with Selenium + Chrome Webdriver in headless mode, and the site was being able to detect there was some automation going on.
Click to expand...

Do not run headless mode--headless mode is very easily detected. A simple test for canvas or opengl will easily detect any headless mode browsers.
 
adverts said:
Do not run headless mode--headless mode is very easily detected. A simple test for canvas or opengl will easily detect any headless mode browsers.
Click to expand...

What's a workaround then if one wants to not see the browser window opened?
Would running everything in Docker be a solution?
 
夢市片 said:
What's a workaround then if one wants to not see the browser window opened?
Would running everything in Docker be a solution?
Click to expand...

Docker doesn't render GUI as far as I know. Containerization is only for console apps.

To emulate a user, you have to behave like a user. No real "user" would ever browse in headless mode. The bottom line is you'll have to run the full browser to behave like a user. Full virtual machines can help you run multiple instances of browsers with a GUI.
 
adverts said:
Docker doesn't render GUI as far as I know. Containerization is only for console apps.

To emulate a user, you have to behave like a user. No real "user" would ever browse in headless mode. The bottom line is you'll have to run the full browser to behave like a user. Full virtual machines can help you run multiple instances of browsers with a GUI.
Click to expand...

It can if you use xfvb iirc.

Alright, that's what I was thinking. A VPS could do the job aswell then.
 
You have it twisted OP. That is like saying should I use this old banger of a car when I have been driving a sportscar.

Like others say selenium is a last resort and for the others saying that 'resource use isn't important' they a idiots. If your bot is 500 times more efficient then it is 500 times the earnings for the same price. And selenium resource use quickly adds up once you get in the double figures of threads. These guys who say that obviously are not proper coders.
 
PineappleCrush said:
You have it twisted OP. That is like saying should I use this old banger of a car when I have been driving a sportscar.

Like others say selenium is a last resort and for the others saying that 'resource use isn't important' they a idiots. If your bot is 500 times more efficient then it is 500 times the earnings for the same price. And selenium resource use quickly adds up once you get in the double figures of threads. These guys who say that obviously are not proper coders.
Click to expand...
I think there is a misunderstanding, I agree with you fully. Using Requests should be the first tool used due to far less resource usage. I've profiled Requests vs Selenium, and we are talking 100X more resource usage with Selenium. Plus, Selenium is cludgy to use, where reading a simple HTTP status code would suffice. I 100% prefer Requests, and would use 100% of the time if there weren't pages that require Javascript either due to rendering or fingerprinting.
 
JackTheRooster said:
I think there is a misunderstanding, I agree with you fully. Using Requests should be the first tool used due to far less resource usage. I've profile Requests vs Selenium, and we are talking 100X more resource usage with Selenium. Plus, Selenium is cludgy to use, where reading a simple HTTP status code would suffice. I 100% prefer Requests, and would use 100% of the time if there weren't pages that require Javascript either due to rendering or fingerprinting.
Click to expand...

Can you name one somewhat popular site that doesn't use Javascript for some sort of telemetry? If you want to emulate a user, you have to parse and handle Javascript in today's world.

If you don't care that your traffic may appear like a bot, go with requests. It really depends on your use case and you have to weight the cost/benefits of resource use vs user-like behavior.
 
adverts said:
Can you name one somewhat popular site that doesn't use Javascript for some sort of telemetry? If you want to emulate a user, you have to parse and handle Javascript in today's world.

If you don't care that your traffic may appear like a bot, go with requests. It really depends on your use case and you have to weight the cost/benefits of resource use vs user-like behavior.
Click to expand...
A general good rule of thumb is using requests for read-only info that is available on the public API. Anything user-based may require browser emulation, but some elements can be done via HTTP. Sure, many sites do send Telemetry data, but they have to take into account legitimate users who block the telemetry sending for privacy reasons. It's an endless game trying to outsmart the botters.
 
夢市片 said:
It can if you use xfvb iirc.

Alright, that's what I was thinking. A VPS could do the job aswell then.
Click to expand...
Yep, xfvb allows virtual rendering of the browser so you don't have to run in headless mode. This was a workaround for a long time before headless came out. However, more resources are consumed, so if you don't need it, avoid it.
 
夢市片 said:
Selenium as a whole is easily detected most of the time. But I heard that you can use MultiLogin with Selenium, and control their own browsers with it.
Click to expand...

Just a clarification, Multilogin embeds Selenium Hardened which is a custom Selenium build that is protected from detection and fingerprinting. It is trivial to detect the usage of vanilla Selenium.
 
夢市片 said:
Selenium as a whole is easily detected most of the time. But I heard that you can use MultiLogin with Selenium, and control their own browsers with it.
Click to expand...
Hi 夢市片, can you please clairfy on Multilogin? Do they host the VMs that the browsers run on, run them remotely, and do you get that as part of the plan? Or do you have to provide the hardware?
 
JackTheRooster said:
Hi 夢市片, can you please clairfy on Multilogin? Do they host the VMs that the browsers run on, run them remotely, and do you get that as part of the plan? Or do you have to provide the hardware?
Click to expand...

I think @BloodyNinja is more than able to answer the question.
You basically provide the hardware, and run isolated browsers with their own fingerprints/configuration.
 
JackTheRooster said:
Hi 夢市片, can you please clairfy on Multilogin? Do they host the VMs that the browsers run on, run them remotely, and do you get that as part of the plan? Or do you have to provide the hardware?
Click to expand...

No hardware is needed, everything is emulated in a browser. You create 100/1000/10000 browser profiles, assign different proxies to each (or use rotating proxies) and it's equal to having the same amount of unique computers.

You can install Multilogin on your computer or server. A single computer may run 20-30 or more browser profiles at once depending on its specs.
 
BloodyNinja said:
No hardware is needed, everything is emulated in a browser. You create 100/1000/10000 browser profiles, assign different proxies to each (or use rotating proxies) and it's equal to having the same amount of unique computers.

You can install Multilogin on your computer or server. A single computer may run 20-30 or more browser profiles at once depending on its specs.
Click to expand...

I know this is akin to asking a barber if you need a haircut, but does Multilogin work on social media sites like FB, Reddit, etc? I imagine botters could find value in spoofing browser fingerprints.

The plans are fairly confusing. There's collaboration plans, automation plans, and a free plan. My use case would be running minimum 50 browsers (threads) simultaneously from Python, so whatever does that the cheapest. Is this a drop-in replacement if you're using Selenium already? Seems to use a custom webdriver/browser setup.

Does it support authed proxies with the credentials in the proxy url string? IP Whitelisting sucks when you have to pay for static IPs.
 
Advertise on BHW
Status
Not open for further replies.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top