[python & selenium] An intro into browser automation

iamcholo said:

Great share @Machairodont
A wonderful wrapper library for Selenium is undetected-chromedriver by ultrafunkamsterdam

https://github.com/ultrafunkamsterdam/undetected-chromedriver
Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)

Click to expand...

Machairodont said:

Thanks guys,

I will extend this intro as soon as I find some time for it.

Click to expand...

iamcholo said:

Great share @Machairodont
A wonderful wrapper library for Selenium is undetected-chromedriver by ultrafunkamsterdam

https://github.com/ultrafunkamsterdam/undetected-chromedriver
Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)

Click to expand...

azucar said:

Thank you very much for sharing, I look forward to the next contribution.


Some time ago I was trying to play with Selenium on OnlyFans but I was afraid of being detected. Could this be a solution? Thanks

Click to expand...

Machairodont said:

Hi there,

I'm a *big* python lover... just that you know

I'm using python and selenium quite for a longtime to automate my stuff (if python itself is not enough). And I want to write here a short howto.

What is Selenium?
It's a browser automation framework. Most often used with Java (yuck) and Python. It needs a brower-driver, supports Chrome and Firefox (and possibly others).

Where to start?
Readthedocs and how to start you can find here: https://selenium-python.readthedocs.io/

Important things to consider

Change user-agent

Python:

from selenium import webdriver
opts = webdriver.ChromeOptions()
opts.add_argument("user-agent=whatever you want")

driver = webdriver.Chrome(options=opts)

Load a chrome extension (very handy!)

Python:

from selenium import webdriver
opts = webdriver.ChromeOptions()
opts.add_extension(<path to crx file>)

driver = webdriver.Chrome(options=opts)

(In one project I'm using Ahrefs Chrome extension to extract some ahrefs stats about sites)

Use a proxy

Python:

from selenium import webdriver
proxy= "12.34.56.78:3456" # put your proxy IP:PORT here
opts = webdriver.ChromeOptions()
opts.add_argument('--proxy-server=%s' % proxy)

driver = webdriver.Chrome(options=opts)

(some basic example how to do a rotating proxy can be found here: https://gist.github.com/tushortz/cba8b25f9d80f584f807b65890f37be5)

Run in headless mode (no browser window visible anymore)

Python:

from selenium import webdriver
opts = webdriver.ChromeOptions()
opts.add_argument("--headless")

driver = webdriver.Chrome(options=opts)

(be carefull with this, it can be detected on the server side!)

Load and Save browser cookies
Save:

Python:

import pickle
from selenium import webdriver

driver = webdriver.Chrome()
driver.get("http://www.yoursite.com")
#login, do some stuff here manually
#then press enter in the shell windows to save the resulting cookie
input()
pickle.dump(driver.get_cookies() , open("cookies.dat","wb"))

Load:

Python:

import pickle
from selenium import webdriver

driver = webdriver.Chrome()
cookies = pickle.load(open("cookies.dat", "rb"))
#restore saved cookies
for cookie in cookies:
    driver.add_cookie(cookie)
driver.get("http://www.yoursite.com")

Type text like a human

Python:

import random
from selenium import webdriver

def human_text_enter(element, text):
    element.click()
    for char in text:
        start = 0.1 #please edit the speed here
        stop = 0.6 #change this (the maximum value is 1 or 0.9)
        step = 0.3
        precision = 0.1
        f = 1 / precision
        n = random.randrange(start * f, stop * f, step * f) / f
        time.sleep(n)
        element.send_keys(char) 
    time.sleep(n)

driver = webdriver.Chrome()
driver.get("http://www.yoursite.com")

Example login to Ahrefs.com

Python:

import random, time
from selenium import webdriver

driver.get("https://app.ahrefs.com/user/login")
time.sleep(3)
ahrefs_username = 'username'
ahrefs_password = 'password\n'
ahref_email_input = driver.find_element_by_css_selector('input[type=email]')
human_text_enter(ahref_email_input, ahrefs_username)
ahref_pw_input = driver.find_element_by_css_selector('input[type=password]')
human_text_enter(ahref_pw_input, ahrefs_password)
time.sleep(3)

Open a new tab

Python:

driver.execute_script("window.open('{}');".format('https://yoursite.com'))

(first tab/page must be already opened!)

The different tabs can be accessed in the window_handle

Python:

handles = driver.window_handles
driver.switch_to.window(handles[0])
driver.switch_to.window(handles[1])

Some advanced stuff
It can make sense to setup different user accounts (for the platform you want to use) with different proxies (and with different selenium sessions of course!), then save the resulting cookies in a file with the proxy IP as name.
Next time you use the same proxy, you load that saved cookies file to restore the session like before.
If the platform has no session timeout (yeah that still exists sometimes!) you don't need to login again.
Even if you need to login again (not manually anymore, but with some lines of python of course),
you don't get nasty "don't know this computer", "don't know this ip" stuff anymore...
Combined with a rotating proxy approach this kind of setup can be very powerful.

Browser fingerprinting is used on more and more sites...
Some info about how this fingerprinting is working:
https://fingerprintjs.com/blog/browser-fingerprinting-techniques/
https://browserleaks.com/fonts
https://privacycheck.sec.lrz.de/active/fp_cf/fp_canvas_font.html
Nice read about mimicking human behavior:
https://www.binarydefense.com/mimicking-human-activity-using-selenium-and-python/
The combination of Browser extensions together with python and selenium opens powerful new possibilities.
To get inspirations, take a read here: https://intoli.com/blog/sandbox-breakout/

Click to expand...

iamcholo said:

it’s depends
Are you behavior like a boot (quick actions) or human (delays)?

Take a look into this repository
https://github.com/DIGITALCRIMINAL/OnlyFans

Click to expand...

HankaRobotics said:

What's the best combination for avoiding anti-bot measures ? Puppeteer extra stealth with Node.js ? Or is there something more effective in python ?

Click to expand...

HankaRobotics said:

What's the best combination for avoiding anti-bot measures ? Puppeteer extra stealth with Node.js ? Or is there something more effective in python ?

Click to expand...