Headless Selenium and Google search. Is it possible?

I have written quite a few selenium python scripts but I wonder if I can run them headless from my ubuntu server and if it can be done without google realising?
If so what are the selenium driver options?

Very difficult ...

Not sure about selenium but Puppeteer have something called Puppeteer Extra and it's A Plugin for Scraping

Use undetected chromedriver and it should work fine,
You might need some proxies and implement captcha solving service, depending in your usage,

Make sure to take your time with testing before scaling,
You don't want to end up with unstable results or losing your time finding out it's getting stuck or crashing somehow

Stripe maker said:

Not sure about selenium but Puppeteer have something called Puppeteer Extra and it's A Plugin for Scraping

Click to expand...

Puppeteer extra and stealth work pretty well. I use Puppeteer if I need access to browser internals like network requests, DOM manipulation, rendering, etc. Unfortunately Selenium doesn't do that so you would need a proxy or some other type of monitoring tool.

Selenium you can set quite a few optimizations that also work well and get around detection.

I use both for different projects and normally run through cloud services as docker instances and use proxies. You can run them off your local, remote, or wherever is convenient for you.

robifz said:

I have written quite a few selenium python scripts but I wonder if I can run them headless from my ubuntu server and if it can be done without google realising?
If so what are the selenium driver options?

Click to expand...

It's possible. Tho you need something to solve the captchas which will be more frequent with a headless browser.

You could just use pure requests, there is no need for a browser simulator. If you encounter a captcha just wait or change proxy

hideath said:

Use undetected chromedriver and it should work fine,
You might need some proxies and implement captcha solving service, depending in your usage,

Make sure to take your time with testing before scaling,
You don't want to end up with unstable results or losing your time finding out it's getting stuck or crashing somehow

Click to expand...

Thanks. I have just started to use undetected chrome driver. Are there a set of recommended options to pass to it?
Currently for the normal chrome driver I have...
chrome_options.add_argument("--incognito")

preferences = {
"webrtc.ip_handling_policy": "disable_non_proxied_udp",
"webrtc.multiple_routes_enabled": False,
"webrtc.nonproxied_udp_enabled": False
}
chrome_options.add_experimental_option("prefs", preferences)

ua = UserAgent()
user_agent = ua.random
print(user_agent)

chrome_options.add_argument(f'--user-agent={user_agent}')

width = random.randint(800, 1600)
height = random.randint(600, 900)
driver.set_window_size(width, height)