Does anyone have an undetectable selenium jar?

I'm running late but if you still need I can dig up the rough code. It's rough proof of concept but it does work.
I'm having some trouble with the Python libs. If you still have some of your rough code, I'd love to see it if you wouldn't mind showing me it.

The few PyChrome libs out there on github seem to be so out of date that they don't even spawn a browser instance.

I believe you can spawn a browser instance through the command prompt and assign a port number:
Code:
start chrome.exe --remote-debugging-port=9222 --user-data-dir=remote-profile

Hooking into the browser instance is what is giving me an issue here. Do you know any specific url or source I can look at for help SEOMadHatter?
 
I'm having some trouble with the Python libs. If you still have some of your rough code, I'd love to see it if you wouldn't mind showing me it.

The few PyChrome libs out there on github seem to be so out of date that they don't even spawn a browser instance.

I believe you can spawn a browser instance through the command prompt and assign a port number:
Code:
start chrome.exe --remote-debugging-port=9222 --user-data-dir=remote-profile

Hooking into the browser instance is what is giving me an issue here. Do you know any specific url or source I can look at for help SEOMadHatter?
Cefpython may been what you need, undetectable by distil
 
I'm having some trouble with the Python libs. If you still have some of your rough code, I'd love to see it if you wouldn't mind showing me it.

The few PyChrome libs out there on github seem to be so out of date that they don't even spawn a browser instance.

I believe you can spawn a browser instance through the command prompt and assign a port number:
Code:
start chrome.exe --remote-debugging-port=9222 --user-data-dir=remote-profile

Hooking into the browser instance is what is giving me an issue here. Do you know any specific url or source I can look at for help SEOMadHatter?

Oh sorry, I forgot about this thread. I have a lib to handle the instance creation but it's heavily tied into my system but I'll spawn the instance as a subprocess with different remote debugging ports. They'll also be given specific profile folders and proxies. If your proxy requires authentication you'll have to load an extension in to set it because it won't support proxy authentication through the command line.

Then connecting it to Selenium is built right in.

Windows example:

chrome_options.add_experimental_option("debuggerAddress", "127.0.0.1:9222")
thechromedriver = webdriver.Chrome("chromedriver.exe", chrome_options=chrome_options)
 
Alright took me a while but here is final solution for anyone interested:


run the following line in command prompt:
Code:
start chrome.exe --remote-debugging-port=5351

then in Python:
Code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_experimental_option("debuggerAddress", "127.0.0.1:5351")  # Note the port numbers should match.
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.get(Distil_Page)

Note that I didn't start up a headless version of Chrome. Distil is somehow detecting that Chrome is headless, while non-headless Chrome works. You can spoof the user agent through command prompt, but they still detect that Chrome is headless, so I just left it. I think there should be a command in Python to minimize the browser window once its created so it should be fine.

I tested CefPython, and it does work, but I like the feel of this better. Thanks for the help everyone. Will be using this to scrape. Also will probably need to assign browser its own profile and proxy through command prompt, but that should be trivial.


Thus concludes my 4 month search of figuring out how to web scrape protected sites in Python.
 
Note that I didn't start up a headless version of Chrome. Distil is somehow detecting that Chrome is headless, while non-headless Chrome works. You can spoof the user agent through command prompt, but they still detect that Chrome is headless, so I just left it. I think there should be a command in Python to minimize the browser window once its created so it should be fine.
...
Thus concludes my 4 month search of figuring out how to web scrape protected sites in Python.

damn, 4 months...
did a 5 minute google search and found this from a year ago: https://intoli.com/blog/not-possible-to-block-chrome-headless/

seems to work for distil
i tried it on www.distilnetworks.com and www.whitepages.com
but you have to use javascript

my test
Code:
// We'll use Puppeteer is our browser automation framework.
const puppeteer = require('puppeteer');

// This is where we'll put the code to get around the tests.
const preparePageForTests = async (page) => {
  // Pass the User-Agent Test.
  const userAgent = 'Mozilla/5.0 (X11; Linux x86_64)' +
    'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.39 Safari/537.36';
  await page.setUserAgent(userAgent);

  // Pass the Webdriver Test.
  await page.evaluateOnNewDocument(() => {
    Object.defineProperty(navigator, 'webdriver', {
      get: () => false,
    });
  });

  // Pass the Chrome Test.
  await page.evaluateOnNewDocument(() => {
    // We can mock this in as much depth as we need for the test.
    window.navigator.chrome = {
      runtime: {},
      // etc.
    };
  });

  // Pass the Permissions Test.
  await page.evaluateOnNewDocument(() => {
    const originalQuery = window.navigator.permissions.query;
    return window.navigator.permissions.query = (parameters) => (
      parameters.name === 'notifications' ?
        Promise.resolve({ state: Notification.permission }) :
        originalQuery(parameters)
    );
  });

  // Pass the Plugins Length Test.
  await page.evaluateOnNewDocument(() => {
    // Overwrite the `plugins` property to use a custom getter.
    Object.defineProperty(navigator, 'plugins', {
      // This just needs to have `length > 0` for the current test,
      // but we could mock the plugins too if necessary.
      get: () => [1, 2, 3, 4, 5],
    });
  });

  // Pass the Languages Test.
  await page.evaluateOnNewDocument(() => {
    // Overwrite the `plugins` property to use a custom getter.
    Object.defineProperty(navigator, 'languages', {
      get: () => ['en-US', 'en'],
    });
  });
}

(async () => {
  // Launch the browser in headless mode and set up a page.
  const browser = await puppeteer.launch({
    args: ['--no-sandbox'],
    headless: true,
  });
  const page = await browser.newPage();

  // Prepare for the tests (not yet implemented).
  await preparePageForTests(page);

  // Navigate to the page that will perform the tests.
  //const testUrl = 'https://www.distilnetworks.com/';
  const testUrl = 'https://www.whitepages.com/';
  await page.goto(testUrl);
  await page.waitFor(3000);

  // Save a screenshot of the results.
  await page.screenshot({path: 'headless-test-result.png'});

  // Clean up.
  await browser.close()
})();
 
damn, 4 months...
did a 5 minute google search

I came across that a while ago. It was one of the first things I found actually. I couldn't figure out how to inject the JS through Python. Also and he's using puppeteer, which is for NodeJS.

NodeJS webdrivers like NightmareJS can get through without any injection at all (someone made me a scraper for Distill a while ago through Nightmare and NodeJS for like $120). It got blocked after like 5 requests, but once restarted would work again. I don't think anyone who knows more than one programming language will have any problem getting through Distill.

Distill seems to have just blocked programmers who basically only use common scraping methods like Selenium with easy to learn languages like Python. I'm sure there's an easy solution in another language, my point is its hard to find a solution in Python, (At least without knowing how to inject JS into a browser).

I left that link alone because I was specifically trying to find a solution to my problem in Python, cause it's a language I could understand and maintain. I have no doubt Puppeter and NightmareJS get through fine. I found out they did early on, I just kept looking for a solution in Python because I didn't have the time to get comfortable w/ a new language.
 
Last edited:
that script only sends some commands via the chrome-devtools-protocol
you only had to translate it to python

like this using pychrome
Code:
import pychrome
import base64

browser = pychrome.Browser(url="http://127.0.0.1:9222")
tab = browser.new_tab()


tab.start()
tab.call_method("Network.enable")
tab.call_method("Page.enable")

tab.call_method("Network.setUserAgentOverride", userAgent="Mozilla/5.0 (X11; Linux x86_64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.39 Safari/537.36" )

scripts = [
"""
(() => {
Object.defineProperty(navigator, 'webdriver', {
  get: () => false,
});
})()
""",
"""
(() => {
// We can mock this in as much depth as we need for the test.
window.navigator.chrome = {
  runtime: {},
  // etc.
};
})()
""",
"""
(() => {
const originalQuery = window.navigator.permissions.query;
return window.navigator.permissions.query = (parameters) => (
  parameters.name === 'notifications' ?
    Promise.resolve({ state: Notification.permission }) :
    originalQuery(parameters)
);
})()
""",
"""
(() => {
// Overwrite the `plugins` property to use a custom getter.
Object.defineProperty(navigator, 'plugins', {
  // This just needs to have `length > 0` for the current test,
  // but we could mock the plugins too if necessary.
  get: () => [1, 2, 3, 4, 5],
});
})()
""",
"""
(() => {
// Overwrite the `plugins` property to use a custom getter.
Object.defineProperty(navigator, 'languages', {
  get: () => ['en-US', 'en'],
});
})()
"""
]
for s in scripts:
    tab.call_method("Page.addScriptToEvaluateOnNewDocument", source=s )
#tab.call_method("Page.navigate", url="https://intoli.com/blog/not-possible-to-block-chrome-headless/chrome-headless-test.html", _timeout=5)
tab.call_method("Page.navigate", url="https://www.whitepages.com", _timeout=5)


tab.wait(2)
png = tab.call_method("Page.captureScreenshot")
with open("screenshot.png", "wb") as fh:
    fh.write(base64.decodestring(png['data']))
    fh.close()
tab.stop()

browser.close_tab(tab)

you have to start a chrome instance first
pychrome doesn't do that for you
Code:
google-chrome  --remote-debugging-port=9222 --headless --no-sandbox
 
sockpuppet Sorry for not replying sooner, I've been busy w/ a few other things. I will try this soon and let you know how it goes. Funny, I've come across Pychrome as well, I just passed it by because I couldn't figure out how to use it. I managed to get as far as to:
browser = pychrome.Browser(<I cant post l - i - n - k - s - on - B - H - W - y- e - t>)
but then I stopped when I got an error. When I tried this months ago I did not realize PyChrome didn't spawn a browser instance for you. I just kept getting the same error and then left after that because there weren't any easy to follow YouTube tutorials on it.

So far trying it again now, I managed to spawn a browser instance and navigate somewhere, but I will do the JS later cause man am I tired. Thank you for all the help man and SEOMadHatter .

I don't think I'm the only one here who's been helped. I feel like there has to be at least like 10 more people lurking on this thread using some of the solutions here.
 
Last edited:
So in theory if we bypass bot detection we could also make a little program that takes proxies and produce views? *puts a black hat on*
 
so anybody figured out how to make selenium undetectalbe?
 
So in theory if we bypass bot detection we could also make a little program that takes proxies and produce views? *puts a black hat on*
I don't get it. You can already do that, why would you require something special? From all I could find there are only a dozen of websites detecting Selenium, and spotify, youtube & whatever don't do it. Did I miss something or are you missing something?
 
I don't get it. You can already do that, why would you require something special? From all I could find there are only a dozen of websites detecting Selenium, and spotify, youtube & whatever don't do it. Did I miss something or are you missing something?
They probably detect it but dont block at small scale
 
Back
Top