[TUTORIAL] Beginners guide to using selenium for python browser automation

pip install selenium

from selenium import web driver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By

driver = webdriver.Firefox()
driver.get("http://www.python.org")
assert "Python" in driver.title
elem = driver.find_element(By.NAME, "q")
elem.clear()
elem.send_keys("pycon")
elem.send_keys(Keys.RETURN)
assert "No results found." not in driver.page_source
driver.close()

python python_org_search.py

# Firefox Selenium four
from selenium import web driver
from selenium.webdriver.firefox.service import Service as FirefoxService
from webdriver_manager.firefox import GeckoDriverManager

driver = webdriver.Firefox(service=FirefoxService(GeckoDriverManager().install()))
# Chrome Selenium four
from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()))

try:
    # Wait 10 seconds before looking for element
    element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, "myDynamicElement"))
    )
finally:
    # Else quit
    driver.quit()

try:
  #Your great code here
except:
  print("An exception occurred")

from selenium import webdriver

from selenium.webdriver.chrome.options import Options

chrome_options = Options()

chrome_options.add_argument("--headless")

driver = webdriver.Chrome(options=chrome_options)

• Evading selenium detection – the ultimate guide

• Playwright guide

• Speed up selenium script development

thebotmaker said:

A useful library I found was called WebDriverManager – this will download the latest version of Selenium whenever you run it. I kid you not, I only discovered this a few months ago. This would have been particularly useful had I found it earlier. here is a code snippet:

Click to expand...

Panther28 said:

Thanks dude, this is incredibly handy to know.

Click to expand...

Jacke met said:

i am trying to automate a social media posting but i want to select all files but make a delay betwen each one of the how i can do it ???

Click to expand...

import time
 
# using sleep() to hault the code execution
time.sleep(6)

thebotmaker said:

Overview

Selenium is an open-source suite of tools for automating web browsers. It supports a variety of programming languages and can be used to automate a wide range of tasks, such as testing web applications, filling out forms, and scraping data from websites. It also has it’s uses as a digital, and in particular black-hat marketer.

For this guide we are focusing on Python, as that is one of the more straightforward languages to get started with.

Despite being frowned upon, Selenium still has it is uses for browser automation. There are sites that do not care or straight up do not detect you are using it depending on what you are doing, and there are ways of hiding the fact you are using this library which we will explore down the line.

A guide like this would have been great when I first got into botting and browser automation specifically – instead, I had to learn for myself. You are in for a treat . Away we go!


General Use Cases – my thoughts added

Web application testing: Selenium can be used to test the functionality and user experience of a web application. This can help ensure that the application is working as expected and improve the user experience for visitors.

Definitely – think of it from the user’s perspective, if you needed to test parts of your sales page e.g., the order form in order to make sure it loads fast and looks great on a variety of different devices.

Data scraping: Using Selenium, you can extract data from websites, such as product details, prices, and reviews. This information can be used to gain insights into the market and to help shape marketing plans. –

Definitely – some websites deploy protection but with the right approach can still be scraped.

Automating social media tasks: Selenium can be used to automate tasks on social media platforms, such as posting updates, liking, and commenting on posts, and following and unfollowing users. This can help save time and improve the efficiency of social media marketing efforts.

Definitely – as said above many sites can detect Selenium, but there are ways to evade this which I will cover in another post.

Email marketing automation: Selenium can be used to automate tasks related to email marketing, such as sending emails, managing email lists, and tracking the performance of email campaigns.

Definitely – email marketing provides automation, so you would not really have to automate this with selenium unless there is a reason.

Search engine optimization (SEO): Selenium can be used to automate tasks related to SEO, such as analysing the performance of websites and identifying potential issues that could impact their ranking in search engine results.

Definitely – it is not talked about a lot, but CTR is a ranking factor you should look out for.

Ad targeting and retargeting: Selenium can be used to automate tasks related to ad targeting and retargeting, such as identifying potential customers and delivering targeted ads to them based on their online behaviour.

Probably – have not investigated this but seems interesting.


Step 1 – Install Python if needed

https://selenium-python.readthedocs.io/installation.html#introduction#


Installation of Python is straightforward if you are on Windows just go to this link:

http://www.python.org/download

Select the version, follow the instructions, and select the option “Add to Path” while installing.


Step 2 – Install Selenium

From the docs:

Use pip to install the selenium package. Python three has pip available in the standard library. Using pip, you can install selenium like this:

Code:

pip install selenium

You may consider using virtualenv to create isolated Python environments. Python three has venv which is the same as virtualenv.

You can also download Python bindings for Selenium from the PyPI page for selenium package. and install manually.


Step 3 – Download the correct Web driver for your system

https://selenium-python.readthedocs.io/installation.html#drivers
After this you will need the right web driver which is linked here:

Chrome:	https://sites.google.com/chromium.org/driver/
Edge:	https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/
Firefox:	https://github.com/mozilla/geckodriver/releases
Safari:	https://webkit.org/blog/6900/webdriver-support-in-safari-10/

Or you can rely on the WebdriverManager which will handle this process for you - more on this in the tips and tricks section towards the end of the post.

Step 4 – Write your first script

https://selenium-python.readthedocs.io/getting-started.html#simple-usage

What you would want to do is paste this into a file, save the and run it.

Code:

from selenium import web driver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By

driver = webdriver.Firefox()
driver.get("http://www.python.org")
assert "Python" in driver.title
elem = driver.find_element(By.NAME, "q")
elem.clear()
elem.send_keys("pycon")
elem.send_keys(Keys.RETURN)
assert "No results found." not in driver.page_source
driver.close()

Assuming you saved the file as “python_org_searchclass.py” and python is in your PATH, you can open a CMD or terminal window in the script directory and call it like this:

Code:

python python_org_search.py

Assuming you did everything correctly, a browser window should have opened, and the script run as expected.
Congratulations on your first selenium script!


Tips and tricks

Use the right version of Selenium: Make sure you are using the right version of Selenium that is compatible with the browser and operating system you are using.

A useful library I found was called WebDriverManager – this will download the latest version of Selenium whenever you run it. I kid you not, I only discovered this a few months ago. This would have been particularly useful had I found it earlier. here is a code snippet:

Code:

# Firefox Selenium four
from selenium import web driver
from selenium.webdriver.firefox.service import Service as FirefoxService
from webdriver_manager.firefox import GeckoDriverManager

driver = webdriver.Firefox(service=FirefoxService(GeckoDriverManager().install()))
# Chrome Selenium four
from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()))

Try / Except (error handling)
When automating a web application, it is common for elements on the page to take different amounts of time to load. To make sure your automation runs smoothly, you can use explicit waits in Selenium to pause your script until a specific element is present on the page before you interact with it.

Code:

try:
    # Wait 10 seconds before looking for element
    element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, "myDynamicElement"))
    )
finally:
    # Else quit
    driver.quit()

Code:

try:
  #Your great code here
except:
  print("An exception occurred")

Use the correct locator strategy: Selenium provides several strategies for locating elements on a page, such as by id, name, class name, tag name, etc.

Make sure you choose the correct locator strategy to ensure that your script can locate the correct element on the page

I will leave this part to the docs as it is more comprehensive: https://selenium-python.readthedocs.io/locating-elements.html


Headless browser:

A headless browser is a great tool for automating web application testing without the need for a graphical user interface. This can help make your Selenium scripts run faster and integrate with other continuous integration environments better.

Code:

from selenium import webdriver

from selenium.webdriver.chrome.options import Options

chrome_options = Options()

chrome_options.add_argument("--headless")

driver = webdriver.Chrome(options=chrome_options)

Use the documentation: The Selenium documentation is a great resource for learning about Selenium and finding solutions to common problems. Make sure to refer to the documentation when you encounter issues with your Selenium scripts – For me personally I use stack overflow as the documentation is easy to grasp at this point.


Resources for learning more

https://www.selenium.dev/documentation/en/
https://www.stackoverflow.com
https://www.geeksforgeeks.org/selenium-python-tutorial/
https://selenium-python.readthedocs.io/
https://www.chris-wells.net/articles/2017/09/01/consistent-selenium-testing/

Next Posts

• Evading selenium detection – the ultimate guide

• Playwright guide

• Speed up selenium script development

Have fun botting and happy holidays!

Click to expand...

thebotmaker said:

Overview

Selenium is an open-source suite of tools for automating web browsers. It supports a variety of programming languages and can be used to automate a wide range of tasks, such as testing web applications, filling out forms, and scraping data from websites. It also has it’s uses as a digital, and in particular black-hat marketer.

For this guide we are focusing on Python, as that is one of the more straightforward languages to get started with.

Despite being frowned upon, Selenium still has it is uses for browser automation. There are sites that do not care or straight up do not detect you are using it depending on what you are doing, and there are ways of hiding the fact you are using this library which we will explore down the line.

A guide like this would have been great when I first got into botting and browser automation specifically – instead, I had to learn for myself. You are in for a treat . Away we go!


General Use Cases – my thoughts added

Web application testing: Selenium can be used to test the functionality and user experience of a web application. This can help ensure that the application is working as expected and improve the user experience for visitors.

Definitely – think of it from the user’s perspective, if you needed to test parts of your sales page e.g., the order form in order to make sure it loads fast and looks great on a variety of different devices.

Data scraping: Using Selenium, you can extract data from websites, such as product details, prices, and reviews. This information can be used to gain insights into the market and to help shape marketing plans. –

Definitely – some websites deploy protection but with the right approach can still be scraped.

Automating social media tasks: Selenium can be used to automate tasks on social media platforms, such as posting updates, liking, and commenting on posts, and following and unfollowing users. This can help save time and improve the efficiency of social media marketing efforts.

Definitely – as said above many sites can detect Selenium, but there are ways to evade this which I will cover in another post.

Email marketing automation: Selenium can be used to automate tasks related to email marketing, such as sending emails, managing email lists, and tracking the performance of email campaigns.

Definitely – email marketing provides automation, so you would not really have to automate this with selenium unless there is a reason.

Search engine optimization (SEO): Selenium can be used to automate tasks related to SEO, such as analysing the performance of websites and identifying potential issues that could impact their ranking in search engine results.

Definitely – it is not talked about a lot, but CTR is a ranking factor you should look out for.

Ad targeting and retargeting: Selenium can be used to automate tasks related to ad targeting and retargeting, such as identifying potential customers and delivering targeted ads to them based on their online behaviour.

Probably – have not investigated this but seems interesting.


Step 1 – Install Python if needed

https://selenium-python.readthedocs.io/installation.html#introduction#


Installation of Python is straightforward if you are on Windows just go to this link:

http://www.python.org/download

Select the version, follow the instructions, and select the option “Add to Path” while installing.


Step 2 – Install Selenium

From the docs:

Use pip to install the selenium package. Python three has pip available in the standard library. Using pip, you can install selenium like this:

Code:

pip install selenium

You may consider using virtualenv to create isolated Python environments. Python three has venv which is the same as virtualenv.

You can also download Python bindings for Selenium from the PyPI page for selenium package. and install manually.


Step 3 – Download the correct Web driver for your system

https://selenium-python.readthedocs.io/installation.html#drivers
After this you will need the right web driver which is linked here:

Chrome:	https://sites.google.com/chromium.org/driver/
Edge:	https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/
Firefox:	https://github.com/mozilla/geckodriver/releases
Safari:	https://webkit.org/blog/6900/webdriver-support-in-safari-10/

Or you can rely on the WebdriverManager which will handle this process for you - more on this in the tips and tricks section towards the end of the post.

Step 4 – Write your first script

https://selenium-python.readthedocs.io/getting-started.html#simple-usage

What you would want to do is paste this into a file, save the and run it.

Code:

from selenium import web driver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By

driver = webdriver.Firefox()
driver.get("http://www.python.org")
assert "Python" in driver.title
elem = driver.find_element(By.NAME, "q")
elem.clear()
elem.send_keys("pycon")
elem.send_keys(Keys.RETURN)
assert "No results found." not in driver.page_source
driver.close()

Assuming you saved the file as “python_org_searchclass.py” and python is in your PATH, you can open a CMD or terminal window in the script directory and call it like this:

Code:

python python_org_search.py

Assuming you did everything correctly, a browser window should have opened, and the script run as expected.
Congratulations on your first selenium script!


Tips and tricks

Use the right version of Selenium: Make sure you are using the right version of Selenium that is compatible with the browser and operating system you are using.

A useful library I found was called WebDriverManager – this will download the latest version of Selenium whenever you run it. I kid you not, I only discovered this a few months ago. This would have been particularly useful had I found it earlier. here is a code snippet:

Code:

# Firefox Selenium four
from selenium import web driver
from selenium.webdriver.firefox.service import Service as FirefoxService
from webdriver_manager.firefox import GeckoDriverManager

driver = webdriver.Firefox(service=FirefoxService(GeckoDriverManager().install()))
# Chrome Selenium four
from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()))

Try / Except (error handling)
When automating a web application, it is common for elements on the page to take different amounts of time to load. To make sure your automation runs smoothly, you can use explicit waits in Selenium to pause your script until a specific element is present on the page before you interact with it.

Code:

try:
    # Wait 10 seconds before looking for element
    element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, "myDynamicElement"))
    )
finally:
    # Else quit
    driver.quit()

Code:

try:
  #Your great code here
except:
  print("An exception occurred")

Use the correct locator strategy: Selenium provides several strategies for locating elements on a page, such as by id, name, class name, tag name, etc.

Make sure you choose the correct locator strategy to ensure that your script can locate the correct element on the page

I will leave this part to the docs as it is more comprehensive: https://selenium-python.readthedocs.io/locating-elements.html


Headless browser:

A headless browser is a great tool for automating web application testing without the need for a graphical user interface. This can help make your Selenium scripts run faster and integrate with other continuous integration environments better.

Code:

from selenium import webdriver

from selenium.webdriver.chrome.options import Options

chrome_options = Options()

chrome_options.add_argument("--headless")

driver = webdriver.Chrome(options=chrome_options)

Use the documentation: The Selenium documentation is a great resource for learning about Selenium and finding solutions to common problems. Make sure to refer to the documentation when you encounter issues with your Selenium scripts – For me personally I use stack overflow as the documentation is easy to grasp at this point.


Resources for learning more

https://www.selenium.dev/documentation/en/
https://www.stackoverflow.com
https://www.geeksforgeeks.org/selenium-python-tutorial/
https://selenium-python.readthedocs.io/
https://www.chris-wells.net/articles/2017/09/01/consistent-selenium-testing/

Next Posts

• Evading selenium detection – the ultimate guide

• Playwright guide

• Speed up selenium script development

Have fun botting and happy holidays!

Click to expand...

bsl19738 said:

Looking for a bot developer. Anyone do that?

Click to expand...

thebotmaker said:

• Evading selenium detection – the ultimate guide

Click to expand...

Panther28 said:

I realized after getting to the stage I'm at that what you posted is the noob of all noob guides to using selenium! lol When you planning on this topic bro? then we really talking about usefulness!

Click to expand...