Does anyone have an undetectable selenium jar?

jamie3000 said:
Anyone got any websites they know for sure are using distil?

I've tried these but I don't think its accurate https://trends.builtwith.com/websitelist/Distil-Bot-Discovery

I've also tried the actual https://www.distilnetworks.com/ website which is obviously using their system but I'm not sure if its actually in full swing or not.

Need some real world websites where people are getting blocked to test on...
Click to expand...
if you're not blocked in first page try this https://www.distilnetworks.com/portal-demo/
have you found any way to evade detection? at least to not flag as selenium
 
Advertise on BHW
Arc717 said:
Alright took me a while but here is final solution for anyone interested:


run the following line in command prompt:
Code: 
start chrome.exe --remote-debugging-port=5351

then in Python:
Code: 
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_experimental_option("debuggerAddress", "127.0.0.1:5351")  # Note the port numbers should match.
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.get(Distil_Page)

Note that I didn't start up a headless version of Chrome. Distil is somehow detecting that Chrome is headless, while non-headless Chrome works. You can spoof the user agent through command prompt, but they still detect that Chrome is headless, so I just left it. I think there should be a command in Python to minimize the browser window once its created so it should be fine.

I tested CefPython, and it does work, but I like the feel of this better. Thanks for the help everyone. Will be using this to scrape. Also will probably need to assign browser its own profile and proxy through command prompt, but that should be trivial.


Thus concludes my 4 month search of figuring out how to web scrape protected sites in Python.
Click to expand...

Can confirm. This gets rid of the navigator.webdriver=true variable. This together with removing the $cdc_ variables in chrome driver, as described here: https://stackoverflow.com/a/52108199

has made google recaptcha 3 a lot easier to solve. Therefore it has circumvented at leas the basic detection. Here's the code I wrote in java to spawn a Webdriver instance in this way:

Code: 
public static WebDriver createChromeDriver(File chromePath, File chromDriver, String proxy) throws IOException {
    int port = getFreePort();
    File profileFolder = new File("selenium/instance" + port);
    if(profileFolder.exists()){
        deleteFileOrDir(profileFolder);
    }
    launchBrowser(chromePath.getAbsolutePath(),
            profileFolder.getAbsolutePath(),
            port, proxy);
    ChromeDriverService builder = new ChromeDriverService.Builder()
            .usingDriverExecutable(chromDriver).build();
    ChromeOptions options = new ChromeOptions();
    options.setExperimentalOption("debuggerAddress", "127.0.0.1:" + port);
    WebDriver driver = new ChromeDriver(builder, options);
    return driver;
}

public static void launchBrowser(String chromePath, String userDir, int port, String proxyServer) throws IOException {
    String proxy = "";
    if(proxyServer != null){
        proxy = "--proxy-server=" + proxyServer;
    }
    String [] params = new String[]{
            chromePath,
            "--remote-debugging-port="+port,
            "--user-data-dir="+userDir,
            proxy
    };
    ProcessBuilder pb = new ProcessBuilder(params);
    Process p = pb.start();
}

public static int getFreePort() throws IOException {
    ServerSocket s = new ServerSocket(0);
    int port = s.getLocalPort();
    s.close();
    return port;
}

public static void deleteFileOrDir(File toDelete) throws IOException {
    if(toDelete.isDirectory()){
        FileUtils.deleteDirectory(toDelete);
    } else {
        FileUtils.deleteQuietly(toDelete);
    }
}
 
xelex said:
Can confirm. This gets rid of the navigator.webdriver=true variable. This together with removing the $cdc_ variables in chrome driver, as described here: https://stackoverflow.com/a/52108199

has made google recaptcha 3 a lot easier to solve. Therefore it has circumvented at leas the basic detection. Here's the code I wrote in java to spawn a Webdriver instance in this way:

Code: 
public static WebDriver createChromeDriver(File chromePath, File chromDriver, String proxy) throws IOException {
    int port = getFreePort();
    File profileFolder = new File("selenium/instance" + port);
    if(profileFolder.exists()){
        deleteFileOrDir(profileFolder);
    }
    launchBrowser(chromePath.getAbsolutePath(),
            profileFolder.getAbsolutePath(),
            port, proxy);
    ChromeDriverService builder = new ChromeDriverService.Builder()
            .usingDriverExecutable(chromDriver).build();
    ChromeOptions options = new ChromeOptions();
    options.setExperimentalOption("debuggerAddress", "127.0.0.1:" + port);
    WebDriver driver = new ChromeDriver(builder, options);
    return driver;
}

public static void launchBrowser(String chromePath, String userDir, int port, String proxyServer) throws IOException {
    String proxy = "";
    if(proxyServer != null){
        proxy = "--proxy-server=" + proxyServer;
    }
    String [] params = new String[]{
            chromePath,
            "--remote-debugging-port="+port,
            "--user-data-dir="+userDir,
            proxy
    };
    ProcessBuilder pb = new ProcessBuilder(params);
    Process p = pb.start();
}

public static int getFreePort() throws IOException {
    ServerSocket s = new ServerSocket(0);
    int port = s.getLocalPort();
    s.close();
    return port;
}

public static void deleteFileOrDir(File toDelete) throws IOException {
    if(toDelete.isDirectory()){
        FileUtils.deleteDirectory(toDelete);
    } else {
        FileUtils.deleteQuietly(toDelete);
    }
}
Click to expand...

Havent gone through the code you posted yet.. Primarily because i suck at Java and usually do python and C#.. But having said that.. Question:
So is your work-around able to make this work headless ?
Would appreciate answer. Thanks.
 
jeflopoim said:
Selenium (outdated)
Puppeteer (the way to go in 2019)

My two cents
Click to expand...
Hey, would you care to elaborate? I
Markus99 said:
Havent gone through the code you posted yet.. Primarily because i suck at Java and usually do python and C#.. But having said that.. Question:
So is your work-around able to make this work headless ?
Would appreciate answer. Thanks.
Click to expand...

Just tried running it headless.
I seem to be getting the same score for google reCaptcha. I get the score from: this page.

Don't know how well it would work in practice though.
 
First thanks--This is wonderful. Referring to post #70 in the thread.
For anyone following along (in Python 3+), you need something akin to the below to make the screenshot--base64.b64decode
Or, at least, I did.
...
tab.wait(2)
png = tab.call_method("Page.captureScreenshot")
imgdata = base64.b64decode(png['data'])
with open("screenshot.png", 'wb') as f:
f.write(imgdata)
tab.stop()
...
FYI, I tried this script on Manta.com...which also works. I'm curious to see how it performs at scale...
Again, thank you. Javascript injection is (definitely) something I'm going woodshed on...
 
Last edited:
Hi guys,

there is a site called tipranks.com which definitely use distil networks. Keep getting detected using selenium and chrome/firefox. If i delete my cache/cookies I get the opportunity to manually respond to a cpatcha but then the site detects again.

Can I ask a really noob question? Is it just selenium that is easily detectable and does anybody know of any other web automation framework/browser combo (preferably controllable via Python) that are not detectable?

btw: here is a good link that may appeal to readers of this thread: https://github.com/dhamaniasad/HeadlessBrowsers
 
dotaboy said:
Can you enlighten me how to spoof navigator.webdriver and $cdc in code?
I don't see where you did that in your code
Click to expand...

I'm pretty sure the, navigator.webdriver gets removed by running a separate chrome process in remote debugging mode, as opposed to letting selenium launch it.

To remove the $cdc variable take a look at the stack overflow link. it's described there.
 
sockpuppet said:
that script only sends some commands via the chrome-devtools-protocol
you only had to translate it to python

like this using pychrome
Code: 
import pychrome
import base64

browser = pychrome.Browser(url="http://127.0.0.1:9222")
tab = browser.new_tab()


tab.start()
tab.call_method("Network.enable")
tab.call_method("Page.enable")

tab.call_method("Network.setUserAgentOverride", userAgent="Mozilla/5.0 (X11; Linux x86_64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.39 Safari/537.36" )

scripts = [
"""
(() => {
Object.defineProperty(navigator, 'webdriver', {
  get: () => false,
});
})()
""",
"""
(() => {
// We can mock this in as much depth as we need for the test.
window.navigator.chrome = {
  runtime: {},
  // etc.
};
})()
""",
"""
(() => {
const originalQuery = window.navigator.permissions.query;
return window.navigator.permissions.query = (parameters) => (
  parameters.name === 'notifications' ?
    Promise.resolve({ state: Notification.permission }) :
    originalQuery(parameters)
);
})()
""",
"""
(() => {
// Overwrite the `plugins` property to use a custom getter.
Object.defineProperty(navigator, 'plugins', {
  // This just needs to have `length > 0` for the current test,
  // but we could mock the plugins too if necessary.
  get: () => [1, 2, 3, 4, 5],
});
})()
""",
"""
(() => {
// Overwrite the `plugins` property to use a custom getter.
Object.defineProperty(navigator, 'languages', {
  get: () => ['en-US', 'en'],
});
})()
"""
]
for s in scripts:
    tab.call_method("Page.addScriptToEvaluateOnNewDocument", source=s )
#tab.call_method("Page.navigate", url="https://intoli.com/blog/not-possible-to-block-chrome-headless/chrome-headless-test.html", _timeout=5)
tab.call_method("Page.navigate", url="https://www.whitepages.com", _timeout=5)


tab.wait(2)
png = tab.call_method("Page.captureScreenshot")
with open("screenshot.png", "wb") as fh:
    fh.write(base64.decodestring(png['data']))
    fh.close()
tab.stop()

browser.close_tab(tab)

you have to start a chrome instance first
pychrome doesn't do that for you
Code: 
google-chrome  --remote-debugging-port=9222 --headless --no-sandbox
Click to expand...

Hey guys, great forum, I'll be sure to stick around.

Line #70, decoding error with base64.decodestring(), try this instead. Works like a treat.

fh.write(base64.b64decode(png['data']))
 
Just letting you know guys that it seems that Instagram has some new way of detecting selenium or bots in general. One day we were able to register empty accounts (with mail only) with a consistent 70% success ratio (30% gave error, please try again later) but now we have 0% success rate. Alredy removed $cdc and did some basic fingerprinting (IP, user agent, plugins, fonts etc). is there currently any strategy to go undetected? Please note we avoided canvas and webgl entirely, as we think there is something specific which triggers the antibot.
 
MarkWilson said:
Just letting you know guys that it seems that Instagram has some new way of detecting selenium or bots in general. One day we were able to register empty accounts (with mail only) with a consistent 70% success ratio (30% gave error, please try again later) but now we have 0% success rate. Alredy removed $cdc and did some basic fingerprinting (IP, user agent, plugins, fonts etc). is there currently any strategy to go undetected? Please note we avoided canvas and webgl entirely, as we think there is something specific which triggers the antibot.
Click to expand...
How about browser window size, initial cursor position?
 
Gogol said:
How about browser window size, initial cursor position?
Click to expand...
window size is already spoofed in order to match a real mobile. didn't know about initial cursor position, will try but I thing something big is going on, all my friends having same problem recently.
 
Hey all, been a while since I was active on the forum. Here are my comments to some of the posts:

@Arc717
> Distil is somehow detecting that Chrome is headless, while non-headless Chrome works.
Yes, got the same experience - I think its related to the checks using Canvas and/or WebGL

Tag me, if you still require an example Python code to launch and control the browser using Chrome Dev Tools protocol (and nothing else, no Selenium).
There are a few nice features within Selenium, but I found everything needed for my job available within Chrome Dev Tools Protocol. Though it is sometimes a pain to deal with.

> CefPython
Can anyone comment on how well does Cef pretend to be a legit Chrome browser against protection scripts? What's the advantage using it over Selenium or Chrome Dev Tools Protocol?

Someone said that Distil focuses on blocking Python because most are is too clueless to use other programming languages. I call this bullshit, who cares what programming language you use to communicate with Chrome? I am using the protocol itself to control Chrome, while the code could be in any language including Pascal.

@DanDD
> so anybody figured out how to make selenium undetectalbe?
Not exactly Selenium, but can use Chrome Dev Tools or other libraries.
 
HelenRyan said:
Hey all, been a while since I was active on the forum. Here are my comments to some of the posts:

@Arc717
> Distil is somehow detecting that Chrome is headless, while non-headless Chrome works.
Yes, got the same experience - I think its related to the checks using Canvas and/or WebGL

Tag me, if you still require an example Python code to launch and control the browser using Chrome Dev Tools protocol (and nothing else, no Selenium).
There are a few nice features within Selenium, but I found everything needed for my job available within Chrome Dev Tools Protocol. Though it is sometimes a pain to deal with.

> CefPython
Can anyone comment on how well does Cef pretend to be a legit Chrome browser against protection scripts? What's the advantage using it over Selenium or Chrome Dev Tools Protocol?

Someone said that Distil focuses on blocking Python because most are is too clueless to use other programming languages. I call this bullshit, who cares what programming language you use to communicate with Chrome? I am using the protocol itself to control Chrome, while the code could be in any language including Pascal.

@DanDD
> so anybody figured out how to make selenium undetectalbe?
Not exactly Selenium, but can use Chrome Dev Tools or other libraries.
Click to expand...
cefpython uses chromium engine so its same as chrome browser
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top