Beginners guide to Multi-Threading in Python: Running Lots of Instances from one PC (mini tutorial)

When you want to run multiple browser instances simultaneously from a single computer, Python offers powerful solutions through its concurrent.futures module and the threading library. Here, we'll explore how to use these tools with two popular browser automation libraries: Playwright and Selenium. This guide will provide code examples for both, helping you harness the power of multi-threading in your projects.

Python is a great for running scraping projects on and developing scripts as its releatively easy for a beginner to pick up.
Other options are Golang and Nodejs which I like as well, becuase Javascript is the language of the web so there's some advantages that I will cover in another guide when we delve into Node Js.

Optimizing your computer for Maximum efficiency
Of course the recommended method is to simply upgrade your hardware. However if you are looking for maximum efficiency...

I'm going to assume your are on Windows, as Linux is already pretty light resource wise so there's no real optimizations I feel are needed.
And of course certain programs will only run on Windows.

For Windows, the basics apply, ensuring that you have no pother programs open, ensure your are using the maximum amount of cores.
You can also change the reservable bandwidth, and reduce startup items to ensure your pgrams get as much power possible, as well as changing the priority.
I'd also look into writing my programs with http requests, of course that guide is coming later.

https://www.dummies.com/article/technology/computers/pcs/set-number-processors-pc-245209/
https://www.thewindowsclub.com/configure-reservable-bandwidth-settings-windows
https://www.pcmag.com/how-to/12-tips-to-speed-up-windows-10-windows-11

1. Setting Up Your Environment​

Before diving into the code, ensure that you have Python installed along with Playwright and Selenium. You can install the necessary libraries using pip:

pip install playwright selenium webdriver-manager


For Playwright, you'll also need to run the installation command for the browsers:

playwright install

2. Multi-Threading with Playwright Using concurrent.futures​

Concurrent uftures is just a way of running a funciton multiple times / with a worker each. Thats what m
Playwright is a modern framework for browser automation. It supports synchronous API which can be beneficial for multi-threading scenarios. Here's how you can use concurrent.futures to run multiple instances of browsers concurrently:

from playwright.sync_api import sync_playwright
from concurrent.futures import ThreadPoolExecutor

def run_browser(url):
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page()
page.goto(url)
print(page.title())
browser.close()

urls = ["<http://example.com>", "<https://google.com>", "<https://github.com>"]

with ThreadPoolExecutor(max_workers=3) as executor:
executor.map(run_browser, urls)


This example creates a function run_browser that each thread should execute, opening a browser, navigating to a URL, printing the page title, and then closing the browser.

3. Multi-Threading with Selenium Using threading.Thread​

Selenium WebDriver is another popular tool for controlling browsers programmatically. Here's how to run multiple instances using the threading.Thread class and webdriver_manager to handle driver binaries:

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from threading import Thread

def run_browser(url):
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get(url)
print(driver.title)
driver.quit()

urls = ["<http://example.com>", "<https://google.com>", "<https://github.com>"]

threads = []
for url in urls:
thread = Thread(target=run_browser, args=(url,))
threads.append(thread)
thread.start()

for thread in threads:
thread.join()



In this script, we define a run_browser function that initializes a Selenium WebDriver for Chrome, navigates to a URL, prints the title of the page, and then closes the driver. Each thread runs this function separately.

Best Practices and Considerations​

• Concurrency Limit: Be aware of the system's limitations. Running too many browser instances can consume significant system resources. So I'd make sure to run a safe number ideally, ensuring I have enough RAM and CPU power to handle it.
• Error Handling: Implement robust error handling within your browser automation functions to manage issues like page load errors or missing elements.
• Resource Management: Ensure browsers are properly closed after operations to free up resources. Definitely necessary. call driver.quit or whatever method is used for closing the browser, as when you arerunning headless browsers they will stay running in the background unless you close them.
• Make sure your software is http request based - its a known fact that the browser is inefficient and slower for writing scripts compared to requests. Wherever possible check if there is an API for whatever site you are trying to automate.
• Thread Safety: Both Playwright and Selenium manage their state well per thread, but always test for thread safety issues, such as shared data that might cause conflicts. never really faced this issue, but something to keep in mind depending on how your are handling your threading

Quick resource list

https://www.trickster.dev/post/you-dont-typically-need-selenium/
https://docs.python-requests.org/en/latest/user/quickstart/#make-a-request
https://www.browserstack.com/guide/playwright-vs-selenium

For next step I would recommend use rest API of a web (if it is possible) or beautifulsoup4