Does anyone have an undetectable selenium jar?

[HelenRyan]

Second this, looking for a working solution against Distil Networks too.
changing $cdc variable inside jar is no longer enough.

 

[SEOMadHatter]

Started digging into this today. Was surprised that a lot of sites haven't done this kind of thing already given how easy the footprints are on the default setting.

I fixed my problem by reverse engineering some of the calls and tapping into the sites API which doesn't seem to give much of a shit where the calls are coming from. But life is easier with Selenium and there's no doubt this kind of thing is going to crop up more and more so I've been poking at it to see if I can get around it if I need to. Made a couple of changes which have gotten me further on a Distil site than the initial default although the check does seem to catch it at a later stage.

It looks at the names of the browser extensions so the Chrome Automation Extension is an obvious footprint. I've disabled it before their check runs and I know I read somewhere earlier today you can hook back into a chromedriver and enable it again if you need the functionality.

Injecting JS before anything else runs I changed the navigator.webdriver to return undefined (not false - a normal Chrome instance would return undefined) and languages to "en-GB", "en-US", "en". Also changed the default resolution and disabled headless (at least for the moment).

Picking through the JS from Distil is slow going because I suck at JS but there's still something catching me out. It's strange since I get by the first check and even though I'm navigating manually the same script running again sounds the alarm when the first didn't. From what I can see all of the responses for the interrogate check would look pretty standard and match a normal Chrome browser. Some JS runs fine and loads data then just dies at a later stage.

 

[HelenRyan]

Started digging into this today. Was surprised that a lot of sites haven't done this kind of thing already given how easy the footprints are on the default setting.

I fixed my problem by reverse engineering some of the calls and tapping into the sites API which doesn't seem to give much of a shit where the calls are coming from. But life is easier with Selenium and there's no doubt this kind of thing is going to crop up more and more so I've been poking at it to see if I can get around it if I need to. Made a couple of changes which have gotten me further on a Distil site than the initial default although the check does seem to catch it at a later stage.

It looks at the names of the browser extensions so the Chrome Automation Extension is an obvious footprint. I've disabled it before their check runs and I know I read somewhere earlier today you can hook back into a chromedriver and enable it again if you need the functionality.

Injecting JS before anything else runs I changed the navigator.webdriver to return undefined (not false - a normal Chrome instance would return undefined) and languages to "en-GB", "en-US", "en". Also changed the default resolution and disabled headless (at least for the moment).

Picking through the JS from Distil is slow going because I suck at JS but there's still something catching me out. It's strange since I get by the first check and even though I'm navigating manually the same script running again sounds the alarm when the first didn't. From what I can see all of the responses for the interrogate check would look pretty standard and match a normal Chrome browser. Some JS runs fine and loads data then just dies at a later stage.

That's indeed sad news - getting caught at a later stage without even using Selenium but having it loaded.

Can you bring some light onto what steps you needed to take apart of navigator.webdriver to undefined to get past the first check?

 

[SEOMadHatter]

That's indeed sad news - getting caught at a later stage without even using Selenium but having it loaded.

Can you bring some light onto what steps you needed to take apart of navigator.webdriver to undefined to get past the first check?

The worst part was I thought I'd cracked it when it got by the block the first time. Proper gut punch after that. I've still got some more playing around to do with it.

I inject javascript before anything else runs which returns undefined on navigator.webdriver instead of false. False works for some tests but not Distil. My hope was to get it looking as close to a normal Chrome browser as possible. The other way is apparently to rebuild Chrome and disable the NavigatorAutomationInformation from firing. I might try that and see if it makes a difference over injecting. This is where life with a laptop sucks. Trying to compile that is going to take an age.

Decoded the hex at the bottom I'm going to go through those variables and see what's being returned. I wasn't kidding when I said I don't know JS so the plan is to rewrite that script a bit to get it running locally and dump out those variables so I can see what's being sent back and see what the differences are vs a normal Chrome browser.

 

[Deleted member 923340]

I am amazed a topic this adavnced is discussed here.

 

[Veil123]

I am amazed a topic this adavnced is discussed here.

It has to be done this way, otherwise google's employees will understand the discussion and we are f*cked.

 

[jamie3000]

https://stackoverflow.com/questions...when-you-are-using-selenium-with-chromedriver

 

[Deleted member 923340]

It has to be done this way, otherwise google's employees will understand the discussion and we are f*cked.

I understand non-Jr VIP members contributed in this thread. But discussions like these should be atleast in Jr.VIp subforum.

I knew about that variable name thingy. If the new version of chromedriver doesnt support it. Why cant we stick with the old version?

 

[Veil123]

I understand non-Jr VIP members contributed in this thread. But discussions like these should be atleast in Jr.VIp subforum.

I knew about that variable name thingy. If the new version of chromedriver doesnt support it. Why cant we stick with the old version?

It was supposed to be in the Jr.VIP so that we don't get tired (like now) finding a solution from the eyes of g0-0gle every single time they change something.

To answer you question. Using an older driver will most likely be a solution for now, but that does not mean that it is the best solution as it will come to a point where using an older version will trigger additional checks.

 

[SEOMadHatter]

I am amazed a topic this adavnced is discussed here.

It probably sounds more complicated than it really is. Automating a browser is colouring by numbers for bot building and luckily their detection has to be done client side so we can see what they're doing. I've no doubt there are people who would have an easy time of this unluckily I'm not a front end dev so it's time-consuming to understand.

https://stackoverflow.com/questions...when-you-are-using-selenium-with-chromedriver

Nothing new there unluckily. Just variations of hexedit away the dc_ regex check but Distil are spotting something else.

I understand non-Jr VIP members contributed in this thread. But discussions like these should be atleast in Jr.VIp subforum.

I knew about that variable name thingy. If for the new version of chromedriver doesnt support it. Why cant we stick to the old version?

There's not much in this thread which hasn't been discussed in public elsewhere.

Touched on this in a post above. Using an older version would get around one check but it's really not ideal since you lose the bug fixes and functionality in later versions. Plus you're running an out of date browser which is never fun. Even if you did use an older version of both Chrome and Chromedriver you'd get around this check but I think Distil are spotting something else at this point. I'll not know for sure until I get the time to rebuild Chrome.

 

[SEOMadHatter]

It was supposed to be in the Jr.VIP so that we don't get tired (like now) finding a solution from the eyes of g0-0gle every single time they change something.

To answer you question. Using an older driver will most likely be a solution for now, but that does not mean that it is the best solution as it will come to a point where using an older version will trigger additional checks.

I guarantee Google engineers would not need this thread. If you're trying to avoid detection from Google them Chromedriver is out. Even Firefox is going to stick a hook in for them if asked. If they start properly going after bots then your only way of running shit is going to wind up OCR and AutoIT.

Luckily third parties like Distil (and other networks like CloudFlare are going to start doing this if it knocks out browser automation so easily) don't have that kind of control over the browser source. All they can do is look for leaks which is fine as long as we can hide them.

 

[Veil123]

I guarantee Google engineers would not need this thread. If you're trying to avoid detection from Google them Chromedriver is out. Even Firefox is going to stick a hook in for them if asked. If they start properly going after bots then your only way of running shit is going to wind up OCR and AutoIT.

Luckily third parties like Distil (and other networks like CloudFlare are going to start doing this if it knocks out browser automation so easily) don't have that kind of control over the browser source. All they can do is look for leaks which is fine as long as we can hide them.

Don't you think it is way easier for them to add more and more "leaks" if they just take a look at the places where that is discussed at most?
Google LOVE bots. Why? Because they learn their AI with the help of botters. How? Just imagine how and why are they using their recaptcha system: Please select a house number? Tell me which one of these is a car/bus? You get my point

 

[Deleted member 923340]

I guarantee Google engineers would not need this thread. If you're trying to avoid detection from Google them Chromedriver is out. Even Firefox is going to stick a hook in for them if asked. If they start properly going after bots then your only way of running shit is going to wind up OCR and AutoIT.

Luckily third parties like Distil (and other networks like CloudFlare are going to start doing this if it knocks out browser automation so easily) don't have that kind of control over the browser source. All they can do is look for leaks which is fine as long as we can hide them.

Finally someone mentioned AutoIT. It is sad no one knows about it.

I have done some GUI automating thing with this. It is so simple yet powerful.

I havent tried automating browsers tho.

 

[SEOMadHatter]

Finally someone mentioned AutoIT. It is sad no one knows about it.

I was kind of half joking to be honest. AutoIT is handy don't get me wrong. I paid my rent with it automating games a few years ago. After that, it was my introduction into scraping and automation a while ago which has made me a bunch of money since.

But it's not really a viable replacement for Selnium and Chromedriver.

Don't you think it is way easier for them to add more and more "leaks" if they just take a look at the places where that is discussed at most?

Even if they had every post on the subject in one place - they wouldn't need to. If they wanted to detect people using Chromedriver it would be candy from a baby.

Distill are limited by existing conditions. Google owns Chrome - they could do whatever the hell they liked with the driver and detect that without running obvious JS checks. Hell, they could just prevent it from accessing the domains on their blacklist.

Google didn't make a move against Chromedriver bots. They changed it to comply with a W3 requirement that automated browsers are clear about being automated browsers. Distill is just picking up on that (and other factors).

 

[Deleted member 923340]

I was kind of half joking to be honest. AutoIT is handy don't get me wrong. I paid my rent with it automating games a few years ago. After that, it was my introduction into scraping and automation a while ago which has made me a bunch of money since.

But it's not really a viable replacement for Selnium and Chromedriver.



Even if they had every post on the subject in one place - they wouldn't need to. If they wanted to detect people using Chromedriver it would be candy from a baby.

Distill are limited by existing conditions. Google owns Chrome - they could do whatever the hell they liked with the driver and detect that without running obvious JS checks. Hell, they could just prevent it from accessing the domains on their blacklist.

Google didn't make a move against Chromedriver bots. They changed it to comply with a W3 requirement that automated browsers are clear about being automated browsers. Distill is just picking up on that (and other factors).

I know there is no better alternative for selenium when it comes to automating browser.

I dont know why you were joking. AutoIT is still used by several enterprises for testing GUIs. (windows antiviruses, installers, players) I personally know a couple.

 

[SEOMadHatter]

I know there is no better alternative for selenium when it comes to automating browser.

I dont know why you were joking. AutoIT is still used by several enterprises for testing GUIs. (windows antiviruses, installers, players) I personally know a couple.

I got started with AutoIT. It is certainly more powerful than the basic macros most people use it for. It's an impressive bit of kit and I'm not taking away from that.

But you said it yourself. It's not a better alternative than Selenium. I was joking because having to literally read the screen with OCR and simulate clicks with AutoIT would be... Less than ideal. It isn't the fault of AutoIT it's the loss of being able to read data from a browser and interact with it without having to literally simulate the click.

That's so unviable for anything serious the suggestion was a joke.

 

[HelenRyan]

Wow that's a lot of posts you've done on Sunday. I appreciate information not being behind the Jr. VIP forum as I've got no access for it.

You mentioned older versions are doing better against detection. What versions/dates are you talking about - 2017? I'd like to compare the two

 

[SEOMadHatter]

Wow that's a lot of posts you've done on Sunday. I appreciate information not being behind the Jr. VIP forum as I've got no access for it.

You mentioned older versions are doing better against detection. What versions/dates are you talking about - 2017? I'd like to compare the two

Not Jr. VIP myself and nothing I've found so far couldn't be found by someone who actually understands JS. The stuff I posted above seems to work in some cases and it's easily applied so worth trying out. If you want to try an older version - anything before October 2017 won't have the flag which identifies it as an automated browser - I'd still apply the other changes though because they're still obvious footprints. The automation extension alone is an obvious one.

If anyone has rebuild Chromium and removed the NavigatorAutomationInformation I'd really appreciate you letting me know how it went. I'm hoping the JS injection isn't working as expected and that'll be the missing part catching me at the later stage - but rebuilding it looks like a task especially since I don't have a desktop.

 

[Bots-Scrapers]

any browser can be fingerprinted and so can selenium hence why its a stupid (and bloated) solution to use a headless browser. people do it just because its easy, but its not the way to go.
raw GET/POST requests is what every professional bot uses. the only excuse to use selenium is "because im too lazy to do it properly"

What? No. Not at all. There are ajax and other js calls that is impossible to investigate and if you emulate via selenium then you'll not face any issue and at least not your action will be flagged.
Sometimes you can not do the http thing nad you have to migrate to a real browser ( not any browser engine i mean ).


Thanks

 

[SEOMadHatter]

What? No. Not at all. There are ajax and other js calls that is impossible to investigate and if you emulate via selenium then you'll not face any issue and at least not your action will be flagged.
Sometimes you can not do the http thing nad you have to migrate to a real browser ( not any browser engine i mean ).


Thanks

This is probably a discussion in itself honestly. There are plenty of spots where it's going to make a lot more sense just to emulate with Selenium than trying to reverse engineer everything. Sure, it's easier. That doesn't mean it's a bad idea.

And if you're reversing calls without rendering JS you still have to look at Distill. Their CDN doesn't settle for just headers that look like a browser. You'll need to reverse their JS before even getting your request through.