Machairodont
Junior Member
- Oct 9, 2021
- 166
- 155
Hi there,
I'm a *big* python lover... just that you know
I'm using python and selenium quite for a longtime to automate my stuff (if python itself is not enough). And I want to write here a short howto.
What is Selenium?
It's a browser automation framework. Most often used with Java (yuck) and Python. It needs a brower-driver, supports Chrome and Firefox (and possibly others).
Where to start?
Readthedocs and how to start you can find here: https://selenium-python.readthedocs.io/
Important things to consider
Change user-agent
Load a chrome extension (very handy!)
(In one project I'm using Ahrefs Chrome extension to extract some ahrefs stats about sites)
Use a proxy
(some basic example how to do a rotating proxy can be found here: https://gist.github.com/tushortz/cba8b25f9d80f584f807b65890f37be5)
Run in headless mode (no browser window visible anymore)
(be carefull with this, it can be detected on the server side!)
Load and Save browser cookies
Save:
Load:
Type text like a human
Example login to Ahrefs.com
Open a new tab
(first tab/page must be already opened!)
The different tabs can be accessed in the window_handle
Some advanced stuff
It can make sense to setup different user accounts (for the platform you want to use) with different proxies (and with different selenium sessions of course!), then save the resulting cookies in a file with the proxy IP as name.
Next time you use the same proxy, you load that saved cookies file to restore the session like before.
If the platform has no session timeout (yeah that still exists sometimes!) you don't need to login again.
Even if you need to login again (not manually anymore, but with some lines of python of course),
you don't get nasty "don't know this computer", "don't know this ip" stuff anymore...
Combined with a rotating proxy approach this kind of setup can be very powerful.
Browser fingerprinting is used on more and more sites...
Some info about how this fingerprinting is working:
https://fingerprintjs.com/blog/browser-fingerprinting-techniques/
https://browserleaks.com/fonts
https://privacycheck.sec.lrz.de/active/fp_cf/fp_canvas_font.html
Nice read about mimicking human behavior:
https://www.binarydefense.com/mimicking-human-activity-using-selenium-and-python/
The combination of Browser extensions together with python and selenium opens powerful new possibilities.
To get inspirations, take a read here: https://intoli.com/blog/sandbox-breakout/
I'm a *big* python lover... just that you know
I'm using python and selenium quite for a longtime to automate my stuff (if python itself is not enough). And I want to write here a short howto.
What is Selenium?
It's a browser automation framework. Most often used with Java (yuck) and Python. It needs a brower-driver, supports Chrome and Firefox (and possibly others).
Where to start?
Readthedocs and how to start you can find here: https://selenium-python.readthedocs.io/
Important things to consider
Change user-agent
Python:
from selenium import webdriver
opts = webdriver.ChromeOptions()
opts.add_argument("user-agent=whatever you want")
driver = webdriver.Chrome(options=opts)Load a chrome extension (very handy!)
Python:
from selenium import webdriver
opts = webdriver.ChromeOptions()
opts.add_extension(<path to crx file>)
driver = webdriver.Chrome(options=opts)Use a proxy
Python:
from selenium import webdriver
proxy= "12.34.56.78:3456" # put your proxy IP:PORT here
opts = webdriver.ChromeOptions()
opts.add_argument('--proxy-server=%s' % proxy)
driver = webdriver.Chrome(options=opts)Run in headless mode (no browser window visible anymore)
Python:
from selenium import webdriver
opts = webdriver.ChromeOptions()
opts.add_argument("--headless")
driver = webdriver.Chrome(options=opts)Load and Save browser cookies
Save:
Python:
import pickle
from selenium import webdriver
driver = webdriver.Chrome()
driver.get("http://www.yoursite.com")
#login, do some stuff here manually
#then press enter in the shell windows to save the resulting cookie
input()
pickle.dump(driver.get_cookies() , open("cookies.dat","wb"))
Python:
import pickle
from selenium import webdriver
driver = webdriver.Chrome()
cookies = pickle.load(open("cookies.dat", "rb"))
#restore saved cookies
for cookie in cookies:
driver.add_cookie(cookie)
driver.get("http://www.yoursite.com")Type text like a human
Python:
import random
from selenium import webdriver
def human_text_enter(element, text):
element.click()
for char in text:
start = 0.1 #please edit the speed here
stop = 0.6 #change this (the maximum value is 1 or 0.9)
step = 0.3
precision = 0.1
f = 1 / precision
n = random.randrange(start * f, stop * f, step * f) / f
time.sleep(n)
element.send_keys(char)
time.sleep(n)
driver = webdriver.Chrome()
driver.get("http://www.yoursite.com")Example login to Ahrefs.com
Python:
import random, time
from selenium import webdriver
driver.get("https://app.ahrefs.com/user/login")
time.sleep(3)
ahrefs_username = 'username'
ahrefs_password = 'password\n'
ahref_email_input = driver.find_element_by_css_selector('input[type=email]')
human_text_enter(ahref_email_input, ahrefs_username)
ahref_pw_input = driver.find_element_by_css_selector('input[type=password]')
human_text_enter(ahref_pw_input, ahrefs_password)
time.sleep(3)Open a new tab
Python:
driver.execute_script("window.open('{}');".format('https://yoursite.com'))The different tabs can be accessed in the window_handle
Python:
handles = driver.window_handles
driver.switch_to.window(handles[0])
driver.switch_to.window(handles[1])Some advanced stuff
It can make sense to setup different user accounts (for the platform you want to use) with different proxies (and with different selenium sessions of course!), then save the resulting cookies in a file with the proxy IP as name.
Next time you use the same proxy, you load that saved cookies file to restore the session like before.
If the platform has no session timeout (yeah that still exists sometimes!) you don't need to login again.
Even if you need to login again (not manually anymore, but with some lines of python of course),
you don't get nasty "don't know this computer", "don't know this ip" stuff anymore...
Combined with a rotating proxy approach this kind of setup can be very powerful.
Browser fingerprinting is used on more and more sites...
Some info about how this fingerprinting is working:
https://fingerprintjs.com/blog/browser-fingerprinting-techniques/
https://browserleaks.com/fonts
https://privacycheck.sec.lrz.de/active/fp_cf/fp_canvas_font.html
Nice read about mimicking human behavior:
https://www.binarydefense.com/mimicking-human-activity-using-selenium-and-python/
The combination of Browser extensions together with python and selenium opens powerful new possibilities.
To get inspirations, take a read here: https://intoli.com/blog/sandbox-breakout/
Last edited:
