[python & selenium] An intro into browser automation

Machairodont said:

Hi there,

I'm a *big* python lover... just that you know

I'm using python and selenium quite for a longtime to automate my stuff (if python itself is not enough). And I want to write here a short howto.

What is Selenium?
It's a browser automation framework. Most often used with Java (yuck) and Python. It needs a brower-driver, supports Chrome and Firefox (and possibly others).

Where to start?
Readthedocs and how to start you can find here: https://selenium-python.readthedocs.io/

Important things to consider

Change user-agent

Python:

from selenium import webdriver
opts = webdriver.ChromeOptions()
opts.add_argument("user-agent=whatever you want")

driver = webdriver.Chrome(options=opts)

Load a chrome extension (very handy!)

Python:

from selenium import webdriver
opts = webdriver.ChromeOptions()
opts.add_extension(<path to crx file>)

driver = webdriver.Chrome(options=opts)

(In one project I'm using Ahrefs Chrome extension to extract some ahrefs stats about sites)

Use a proxy

Python:

from selenium import webdriver
proxy= "12.34.56.78:3456" # put your proxy IP:PORT here
opts = webdriver.ChromeOptions()
opts.add_argument('--proxy-server=%s' % proxy)

driver = webdriver.Chrome(options=opts)

(some basic example how to do a rotating proxy can be found here: https://gist.github.com/tushortz/cba8b25f9d80f584f807b65890f37be5)

Run in headless mode (no browser window visible anymore)

Python:

from selenium import webdriver
opts = webdriver.ChromeOptions()
opts.add_argument("--headless")

driver = webdriver.Chrome(options=opts)

(be carefull with this, it can be detected on the server side!)

Load and Save browser cookies
Save:

Python:

import pickle
from selenium import webdriver

driver = webdriver.Chrome()
driver.get("http://www.yoursite.com")
#login, do some stuff here manually
#then press enter in the shell windows to save the resulting cookie
input()
pickle.dump(driver.get_cookies() , open("cookies.dat","wb"))

Load:

Python:

import pickle
from selenium import webdriver

driver = webdriver.Chrome()
cookies = pickle.load(open("cookies.dat", "rb"))
#restore saved cookies
for cookie in cookies:
    driver.add_cookie(cookie)
driver.get("http://www.yoursite.com")

Type text like a human

Python:

import random
from selenium import webdriver

def human_text_enter(element, text):
    element.click()
    for char in text:
        start = 0.1 #please edit the speed here
        stop = 0.6 #change this (the maximum value is 1 or 0.9)
        step = 0.3
        precision = 0.1
        f = 1 / precision
        n = random.randrange(start * f, stop * f, step * f) / f
        time.sleep(n)
        element.send_keys(char) 
    time.sleep(n)

driver = webdriver.Chrome()
driver.get("http://www.yoursite.com")

Example login to Ahrefs.com

Python:

import random, time
from selenium import webdriver

driver.get("https://app.ahrefs.com/user/login")
time.sleep(3)
ahrefs_username = 'username'
ahrefs_password = 'password\n'
ahref_email_input = driver.find_element_by_css_selector('input[type=email]')
human_text_enter(ahref_email_input, ahrefs_username)
ahref_pw_input = driver.find_element_by_css_selector('input[type=password]')
human_text_enter(ahref_pw_input, ahrefs_password)
time.sleep(3)

Open a new tab

Python:

driver.execute_script("window.open('{}');".format('https://yoursite.com'))

(first tab/page must be already opened!)

The different tabs can be accessed in the window_handle

Python:

handles = driver.window_handles
driver.switch_to.window(handles[0])
driver.switch_to.window(handles[1])

Some advanced stuff
It can make sense to setup different user accounts (for the platform you want to use) with different proxies (and with different selenium sessions of course!), then save the resulting cookies in a file with the proxy IP as name.
Next time you use the same proxy, you load that saved cookies file to restore the session like before.
If the platform has no session timeout (yeah that still exists sometimes!) you don't need to login again.
Even if you need to login again (not manually anymore, but with some lines of python of course),
you don't get nasty "don't know this computer", "don't know this ip" stuff anymore...
Combined with a rotating proxy approach this kind of setup can be very powerful.

Browser fingerprinting is used on more and more sites...
Some info about how this fingerprinting is working:
https://fingerprintjs.com/blog/browser-fingerprinting-techniques/
https://browserleaks.com/fonts
https://privacycheck.sec.lrz.de/active/fp_cf/fp_canvas_font.html
Nice read about mimicking human behavior:
https://www.binarydefense.com/mimicking-human-activity-using-selenium-and-python/
The combination of Browser extensions together with python and selenium opens powerful new possibilities.
To get inspirations, take a read here: https://intoli.com/blog/sandbox-breakout/

Click to expand...

davidsmurf said:

I never managed to pass the WebDriver test here: https://intoli.com/blog/not-possible-to-block-chrome-headless/chrome-headless-test.html

Click to expand...

from selenium import webdriver
opts = webdriver.ChromeOptions()
opts.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.93 Safari/537.36")
opts.add_argument('--disable-blink-features=AutomationControlled')
driver = webdriver.Chrome(options=opts)
driver.get("https://intoli.com/blog/not-possible-to-block-chrome-headless/chrome-headless-test.html")

y00gya said:

can create multiple google accounts with selenium?

Click to expand...

frarg said:

python + selenium, my ultimate combo to scrap

Click to expand...

Machairodont said:

well it depends! To be honest I prefer python and scrapy for the tasks where the anti-bot tec is low enough on the target.
Scrapy is worth a look! It comes out of the box with multi threading and it's unbelievable fast!

check it out here:
https://docs.scrapy.org/en/latest/intro/overview.html

If I find time I maybe could even write some small intro thread here (but seems I will be quite busy the next few months)...
Anyway google search is your friend, there are tons of examples out there for scrapy.

Cheers

Click to expand...

Click to expand...