Headless Selenium and Google search. Is it possible?

[robifz]

I have written quite a few selenium python scripts but I wonder if I can run them headless from my ubuntu server and if it can be done without google realising?
If so what are the selenium driver options?

 

[Cloaked Gambler]

Very difficult ...

 

[Stripe maker]

Not sure about selenium but Puppeteer have something called Puppeteer Extra and it's A Plugin for Scraping

 

[hideath]

Use undetected chromedriver and it should work fine,
You might need some proxies and implement captcha solving service, depending in your usage,

Make sure to take your time with testing before scaling,
You don't want to end up with unstable results or losing your time finding out it's getting stuck or crashing somehow

 

[frozen_yoda]

Not sure about selenium but Puppeteer have something called Puppeteer Extra and it's A Plugin for Scraping

Puppeteer extra and stealth work pretty well. I use Puppeteer if I need access to browser internals like network requests, DOM manipulation, rendering, etc. Unfortunately Selenium doesn't do that so you would need a proxy or some other type of monitoring tool.

Selenium you can set quite a few optimizations that also work well and get around detection.

I use both for different projects and normally run through cloud services as docker instances and use proxies. You can run them off your local, remote, or wherever is convenient for you.

 

[Ashk881]

I have written quite a few selenium python scripts but I wonder if I can run them headless from my ubuntu server and if it can be done without google realising?
If so what are the selenium driver options?

It's possible. Tho you need something to solve the captchas which will be more frequent with a headless browser.

 

[serponaut]

You could just use pure requests, there is no need for a browser simulator. If you encounter a captcha just wait or change proxy

 

[robifz]

Use undetected chromedriver and it should work fine,
You might need some proxies and implement captcha solving service, depending in your usage,

Make sure to take your time with testing before scaling,
You don't want to end up with unstable results or losing your time finding out it's getting stuck or crashing somehow

Thanks. I have just started to use undetected chrome driver. Are there a set of recommended options to pass to it?
Currently for the normal chrome driver I have...
chrome_options.add_argument("--incognito")

preferences = {
"webrtc.ip_handling_policy": "disable_non_proxied_udp",
"webrtc.multiple_routes_enabled": False,
"webrtc.nonproxied_udp_enabled": False
}
chrome_options.add_experimental_option("prefs", preferences)

ua = UserAgent()
user_agent = ua.random
print(user_agent)

chrome_options.add_argument(f'--user-agent={user_agent}')

width = random.randint(800, 1600)
height = random.randint(600, 900)
driver.set_window_size(width, height)