Does anyone have an undetectable selenium jar?

[SEOMadHatter]

Picked through a bit more of it looking at the response which is sent back. webdriver.navigator isn't actually what Distill looks for it's !!webdriver.navigator so make sure that's coming back false. I also found window.innerHeight was a little different than expected. Injection seems to solve both of those.

The only difference in reply is the proof string. Originally I'd thought that was just like an MD5 check to make sure you weren't screwing with the response on a proxy level but since it's the only thing which doesn't match up I wonder if there's more to it than that.

 

[SEOMadHatter]

Last update because I really can't look at code any more today.

It seems that the first response to the JS matches what my normal browser would do. The JS injection changes everything and the only thing which is different is that proof string - which is randomised and I don't think it holds anything important.

What I did notice is the script calls itself repeatedly (I'm assuming this is the same on other Distil sites) and the payload response on the second has somehow gone back to the chromedriver defaults including showing webdirver as true. Not entirely sure how that's a thing since there's not a way to undo Object.defineProperty as far as I can see but it's apparently happening.

 

[SEOMadHatter]

A little more digging and I thought I saw where the injection was being caught out. Object.getOwnPropertyDescriptors(Navigator.prototype).webdriver is still an obvious footprint which a normal browser doesn't have so injecting just for navigator.webdriver doesn't cut it.

A little injection seems to deal with that but I'm still getting flagged up. Somewhere.

Going to have to really improve my reverse engineering skills for this one. And learn JS.

 

[nextime]

any browser can be fingerprinted and so can selenium hence why its a stupid (and bloated) solution to use a headless browser. people do it just because its easy, but its not the way to go.
raw GET/POST requests is what every professional bot uses. the only excuse to use selenium is "because im too lazy to do it properly"

Absolutely agreed, in most cases raw requests are working perfect and no complaints from that. They are using much less memory and working extremely fast, more opportunities to scale the work.

This forum is so much misinformation. How are you going to load recapatcha or other javascript heavy websites? RAW GET/POST requests? Or you are going to reverse engineer the entire javascript for each site you plan on automating to convert them to raw post/get requests? Everyone is a cheap freeloader here I am sure they will be happy to pay some developer a months worth of salary for doing that. You have no idea what you are talking about or why selenium is used.

About recaptcha. Special anti-captcha services provides opportunities to crack recaptcha through their API, you need just send special variables scraped from target web site JS source code and web site URL and then will get result.

About JavaScript heavy websites I know these approaches:

• In most cases you doesn't care about inflated JS because you are just sniffing final get/post request JS performed. There are nothing special and request looks extremely clear, you need just send this request from your own bot with you own parameters, no need to emulate JS.
• Some sites requires JS emulation, you can emulate these parts in your code without using JS engine for this work, it not a big problem in most of cases I have ever seen (I'm professional developer of bots)
• Some sites like instagram have very huge JS inflated logic and it's very difficult to use previous options to handle this shit, in this case bot maker should sniff the official android or iOS app to extract hidden API and use it inside own bot. For example there are many ready libraries for instagram, you can just include it to your project and use that.

In my opinion Selenium is great choice when you have very small workload and enough time to wait, it's faster to make selenium bot, but in the world of bots very often people care about speed/stability/scalability, so raw requests are used more often because they have too important advantages for business.

 

[SEOMadHatter]

Absolutely agreed, in most cases raw requests are working perfect and no complaints from that. They are using much less memory and working extremely fast, more opportunities to scale the work.



About recaptcha. Special anti-captcha services provides opportunities to crack recaptcha through their API, you need just send special variables scraped from target web site JS source code and web site URL and then will get result.

About JavaScript heavy websites I know these approaches:

• In most cases you doesn't care about inflated JS because you are just sniffing final get/post request JS performed. There are nothing special and request looks extremely clear, you need just send this request from your own bot with you own parameters, no need to emulate JS.
• Some sites requires JS emulation, you can emulate these parts in your code without using JS engine for this work, it not a big problem in most of cases I have ever seen (I'm professional developer of bots)
• Some sites like instagram have very huge JS inflated logic and it's very difficult to use previous options to handle this shit, in this case bot maker should sniff the official android or iOS app to extract hidden API and use it inside own bot. For example there are many ready libraries for instagram, you can just include it to your project and use that.

In my opinion Selenium is great choice when you have very small workload and enough time to wait, it's faster to make selenium bot, but in the world of bots very often people care about speed/stability/scalability, so raw requests are used more often because they have too important advantages for business.

I agree with this - mostly.

I'll go low level when it comes to scaling or if ever I was doing a public bot. But, as you pointed out, some sites have pretty insane JS and reverse engineering an app isn't always an option. Sure, it's probably technically possible to build the whole thing reversing JS but... Jesus.

So while low level requests are the ideal I don't think we should be dismissing the idea of using Selenium as the wrong way to do things or just done because lazy. Especially if it's a back-end system then it can be a lot more feasible to develop and maintain with a selenium browser, especially if the bottleneck is external.

There's money to be made and time to be saved with automated browsers. And it's not always feasible or necessary to take reverse everything.

Back on topic:

Someone (it was either here or in a PM I'm way too tired to read through honestly) mentioned the proof hash having data giving it away. It doesn't. It has the script path and ID and the
ajax_header value and uses the time to create a hash to make sure you've actually executed the JS.

So the JS injection literally replies with an 100% match to a normal browser - I just need to figure out how to stop it somehow reverting from that the second time. I really don't want to resort to trying to compile Chromium and I'd have to do it for both Windows and Linux any time I want to update which adds gas to that reluctance.

 

[SEOMadHatter]

With the hash not mattering in theory injecting on the proxy level would do the trick. Setup an MITM proxy and had it swap the responses. So the proof hash is still accurate but it's going to pass like a normal browser on any request - not just the first.

The response seemed to replace fine - but if I'm understanding what I'm looking at Distill had a problem with the SSL cert. I know... nothing about SSL certs so I've no idea how possible that is to get around.

 

[HelenRyan]

Absolutely agreed, in most cases raw requests are working perfect and no complaints from that. They are using much less memory and working extremely fast, more opportunities to scale the work.



About recaptcha. Special anti-captcha services provides opportunities to crack recaptcha through their API, you need just send special variables scraped from target web site JS source code and web site URL and then will get result.

About JavaScript heavy websites I know these approaches:

• In most cases you doesn't care about inflated JS because you are just sniffing final get/post request JS performed. There are nothing special and request looks extremely clear, you need just send this request from your own bot with you own parameters, no need to emulate JS.
• Some sites requires JS emulation, you can emulate these parts in your code without using JS engine for this work, it not a big problem in most of cases I have ever seen (I'm professional developer of bots)
• Some sites like instagram have very huge JS inflated logic and it's very difficult to use previous options to handle this shit, in this case bot maker should sniff the official android or iOS app to extract hidden API and use it inside own bot. For example there are many ready libraries for instagram, you can just include it to your project and use that.

In my opinion Selenium is great choice when you have very small workload and enough time to wait, it's faster to make selenium bot, but in the world of bots very often people care about speed/stability/scalability, so raw requests are used more often because they have too important advantages for business.

Be my guest, share your results in getting around Distil using simple GET/POST requests.

The Topic here is how we can prevent Distil marking requests as Bot activity.

 

[HelenRyan]

With the hash not mattering in theory injecting on the proxy level would do the trick. Setup an MITM proxy and had it swap the responses. So the proof hash is still accurate but it's going to pass like a normal browser on any request - not just the first.

The response seemed to replace fine - but if I'm understanding what I'm looking at Distill had a problem with the SSL cert. I know... nothing about SSL certs so I've no idea how possible that is to get around.

That hash you are talking about is far not MD5. It's using mathematical problem to consume CPU power. What that "proof of work" does is ensures browser is JS enabled and enough CPU power is provided. This method is to ensure mass bots are computationally expensive to be made. Though it is currently not my main concern.

Very interesting find regarding webdriver property getting overriden. I am now trying a different approach - using Remote Chrome console to automate and see if Distil detects automation.
As a simple use-case I've done a Chrome extension that does automation - no detection so far.

 

[SEOMadHatter]

That hash you are talking about is far not MD5. It's using mathematical problem to consume CPU power. What that "proof of work" does is ensures browser is JS enabled and enough CPU power is provided. This method is to ensure mass bots are computationally expensive to be made. Though it is currently not my main concern.

Very interesting find regarding webdriver property getting overriden. I am now trying a different approach - using Remote Chrome console to automate and see if Distil detects automation.
As a simple use-case I've done a Chrome extension that does automation - no detection so far.

Interesting, though I can't imagine that's the bottleneck on most scrapers. I mean if that's your bottleneck you're probably bordering on a DOS attack. The end result is still the same, if the hash isn't right you get a 200 response (if I remember rightly) but you don't get the set cookie headers.

It looks like you could get by Distill without Selenium - just recreate that hash and post back the results it would normally see from a browser.

Interesting idea using remote Chrome I'd be interested to hear how that goes. It's not ideal but I'd imagine it works since it won't trigger as a webdriver. It doesn't cause an obvious extension to load does it?

 

[SEOMadHatter]

You were right about devtools it doesn't set the navigator flag. So Python can hook into it (libs like PyChrome make that a little easier) or Selenium can connect to the devtools port directly. Spawning the driver has to be handled a little differently but the result is the same and gets around Distill. Cool.

 

[intenso09]

I acutally believe that reCaptcha v3 is also detecting Selenium. You can compare it with your regular chrome spam-score here. Would be interesting to know if you got past that somehow.
https://recaptcha-demo.appspot.com/recaptcha-v3-request-scores.php

 

[SEOMadHatter]

I acutally believe that reCaptcha v3 is also detecting Selenium. You can compare it with your regular chrome spam-score here. Would be interesting to know if you got past that somehow.
https://recaptcha-demo.appspot.com/recaptcha-v3-request-scores.php

The same fix would probably sort that unless they go looking for an open devtools port.

 

[HelenRyan]

You were right about devtools it doesn't set the navigator flag. So Python can hook into it (libs like PyChrome make that a little easier) or Selenium can connect to the devtools port directly. Spawning the driver has to be handled a little differently but the result is the same and gets around Distill. Cool.

Yes, I managed to make a few scripts with devtools using PyChrome and it didn't trigger Distil.

Can you share what you've done to avoid Selenium detection using devtools protocol?

 

[HelenRyan]

I acutally believe that reCaptcha v3 is also detecting Selenium. You can compare it with your regular chrome spam-score here. Would be interesting to know if you got past that somehow.

As @SEOMadHatter said, can you try using devtools alone and see if it passes?

 

[PersistentKiwi]

I am amazed a topic this adavnced is discussed here.

It is getting really interesting.

Finally someone mentioned AutoIT. It is sad no one knows about it.

I have done some GUI automating thing with this. It is so simple yet powerful.

I havent tried automating browsers tho.

It would be great to use a normal browser and be able to read HTML and select elements from AutoIT. Not so great resources wise tough.

I acutally believe that reCaptcha v3 is also detecting Selenium. You can compare it with your regular chrome spam-score here. Would be interesting to know if you got past that somehow.
https://recaptcha-demo.appspot.com/recaptcha-v3-request-scores.php

I didn't know that tool, it's great to test bots agains't recaptcha.

 

[Arc717]

I've been lurking on this thread for a few months now, and it has been an interesting to watch. How long does it usually take to find a new solution to things like this once Distil catches on? Im trying to find a stable solution to scrape about 50 urls / second on a Distil site. Is this realistically feasible with PyChrome?

 

[SEOMadHatter]

I've not seen Distill update since I first looked at them so it really depends on what they do. Scraping 50 pages a second with browser emulation is feasible... not neat, but feasible. The programmer in me says reverse engineer the calls. The marketer says if you're making more than you spend on the server who gives a shit.

DevTools works absolutely fine with Distill since you look like a normal browser. No idea how PyChrome works so you'd have to try and see for yourself.

I'd forgotten about this thread. At one point I need to go through that proof of work function and figure out how it's calculated.

 

[Arc717]

I'm not after neat. As long as I can run the bot and not have it use all 16 gigabytes of ram on my PC, I am in the clear. Been awhile since I've learned a new trick in Python. Oh boy.

Btw, I have no experience in JavaScript whatsoever. I have coded extremely inefficient scripts with selenium on sites that don't have any bot blocking simply because I don't know how to reverse the JS.

As for PyChrome, that threw an instant error two lines into it. It does not seem to be as well documented out there as other modules, but I think I will find something on it soon.

From what I've seen on employee reviews of Distill on glass door and other review sites, they aren't pretty. The company appears to be stagnating and going downhill. I'm thinking whatever site I'm after will not up their bot defenses if they are on Distill.

Were you by any chance able to get a working solution with Selenium? I'd be willing to pay through Paypal to not have to go through the hassle. If you haven't though, PyChrome it is.

 

[SEOMadHatter]

Distill claims to be using machine learning on the back end to detect botting patterns but from what I've seen so far it's not great. Unless they update in batches or something. I let a spider run through an obvious single proxy with a lot of requests to see how quickly they'd pick it up and it ran for hours before I switched it off.

The browser detection could be easily adopted by other CDN's though. CloudFlare (for example) already offer anti-botting they could include the same kind of check and block standard Chromedriver instances.

I've not had to use it yet (found an exposed API Distil wasn't properly checking in this projects case) but the proof of concept code basically spawns a chrome instance with subprocess with the right proxy and a fresh profile (which it later deletes) and returns a port number. A selenium instance can then connect to that devtools port and work as normal. You might need to load the automation plugin depending on what you're doing (although keep in mind Distill will see the name of the plugin so if I had to do that I'd look at modifying the name).

I'm running late but if you still need I can dig up the rough code. It's rough proof of concept but it does work.

 

[Arc717]

I know a guy who told me he scrapes 50 urls a second on a Distill site with just one proxy, I thought it was too good to be true.

NightMareJS works for getting through Distill's site. I paid someone on BHW for a scraper, only they did it in NodeJS and it wasn't very fast. Distill definitely does not check certain API. Distill just cracks down hard on Python and selenium since most scrapers use that and don't know how to inject JS and do much more besides: driver.get(url). They repel most bots that way. I do not think they look at IP address or number of requests per second at all, based off of what you just told me about your spider.

I'd love to see your code if you wouldn't mind showing me it. Do you have anything in Python?