Need Help From Python Experts

Does anyone know where to start ? Ive done imacros and stuff but think this is a whole other level lol.

Where can i learn about python guys ?

I need help how i select an element on page.

Here is the python website: https://www.python.org/ I would select add to path when installing.
You can select elements with selenium: https://selenium-python.readthedocs.io/ - Just follow the installation guide and you will be fine.
Also YouTube has some helpful videos for getting started: https://www.youtube.com/results?search_query=getting+started+with+selenium+python+
Best of luck.

If you want to really learn go for https://www.codecademy.com/

Your question is very vague for my understanding but i assume you are talking about selecting a div element in a web page and getting the insides of that div into a python variable.
There are some python scrapper library that do this by default like Beautiful soup 4 and others but i like to use regex (regular expression) you can watch any YouTube videos to learn how to use regex in python.
And you can also use regex101 website to work with regex and practice.

Edit: you can use the Request library in python to get the html of a webpage

There is several library that can help you automate stuffs in website.

For example there is beautifulsoup, Selenium and Pyautogui

Each use in different scenarios. If you want to scrape a regular site that does not use too many js you can use beautifulsoup . Beautifulsoup can be used with requests library. You need to use html element and their classes to get its content.

If you use a site that havily rely on js then your next option is to use Selenium. Using selenium will popup a separate browser window that will act like a regular browser and complete your given task. You can use Xpath of your desire element and actions that selenium will do e.g: clicking items, completing form data etc. You can use a combination of Beautifulsoup and Selenium to get the most out of it.


Lastly PyAutoGui . This library works with screen position. You can use this library to read screen, taking screenshot, clicking content, sending keystrokes on given position on your screen. Use this when something is not clickable or isn't working with Selenium but you can see it on the same place everytime .Just get the screen position of that fed it to pyautogui and set proper action for that item.

@Blank_Hossain those are good suggestions, but...... if he's completely new to programming he ought to start with the basics. Just like we all did calculating the temperature - Fahrenheit to Celsius etc. when he knows the basics, move on to automating libraries

As for tutorials, I would recommend the youtube channel "Programming with Mosh". He his some premium stuffs, but most of his videos are free and open for all.

As for scripts, start with selenium. pyautogui would be a bit stiff for you right now, I guess (but do give it a try).

Here, let me provide you a template to get started with selenium (find_elements_by_xpath is the answer to your question "how to select an element"):

It's a crappy attempt to scrape phone number from an IG profile, but it should be good enough for a start..

By the way, installing python; selenium and the web drivers (chromdriver/firefoxdriver) is 60% of the battle. Once you have everything setup properly, python is like English language.

Another mistake that many newbies (including me when I started) make is not installing python via venv. Doing it via venv will make sure that you can use both python 2.x and 3.x at the same time, instead of having to install them in two separate machines.

Oh, and editors... I suggest using VS Code or pycharm for the editor.

Is it possible with selenium not to use the chrome or firefox web driver but insted to use the real chrome or firefox app?

This is all you need OP, trust: https://www.coursera.org/learn/python

narkomani said:

Is it possible with selenium not to use the chrome or firefox web driver but insted to use the real chrome or firefox app?

Click to expand...

AFAIK, no.

narkomani said:

Is it possible with selenium not to use the chrome or firefox web driver but insted to use the real chrome or firefox app?

Click to expand...

It is possible to load custom browser profiles, if this it what you are asking. But it is still using the webdriver behind the scenes for automation.
See this for example: https://stackoverflow.com/questions/37247336/selenium-use-of-firefox-profile

@Blank_Hossain If we use PyAutoGui with chrome for example, what would be easiest way to get source code of html page?

black-arrow said:

@Blank_Hossain If we use PyAutoGui with chrome for example, what would be easiest way to get source code of html page?

Click to expand...

PyAutoGui is only used with Selenium and Beautifulsoup. Pyautogui currently don't have any function that will give you the source code. But if you just need the source and you are stuck with PyAutoGui then you could try implementing something like right mouse click > Save as html but that's an ugly way of doing it. My suggestion will be using selenium with bs4.

Gogol said:

As for tutorials, I would recommend the youtube channel "Programming with Mosh". He his some premium stuffs, but most of his videos are free and open for all.

As for scripts, start with selenium. pyautogui would be a bit stiff for you right now, I guess (but do give it a try).

Here, let me provide you a template to get started with selenium (find_elements_by_xpath is the answer to your question "how to select an element"):

It's a crappy attempt to scrape phone number from an IG profile, but it should be good enough for a start..

Code:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
from selenium.webdriver.chrome.options import Options
import re
url = input("Enter the URL you want to scrape phone number (only searches in bio:\n")

options = Options()
options.set_headless(headless=True)
options.add_argument('--disable-extensions')
options.add_argument('--hide-scrollbars')
options.add_argument('--disable-gpu')
options.add_experimental_option('excludeSwitches', ['enable-logging'])
driver = webdriver.Chrome(options=options)

print("Opening URL .... \n")
driver.get(url)
print("sleep for 3 seconds (or implement document ready for better results)\n")
time.sleep(3)
description = driver.find_elements_by_xpath('//*[@id="react-root"]/section/main/div/header/section/div[2]')[0].get_attribute("innerText")
m = re.findall('3\d{9}', description)[0]
driver.quit()
print("Scraped Phone number is: ")
print(m)

By the way, installing python; selenium and the web drivers (chromdriver/firefoxdriver) is 60% of the battle. Once you have everything setup properly, python is like English language.

Another mistake that many newbies (including me when I started) make is not installing python via venv. Doing it via venv will make sure that you can use both python 2.x and 3.x at the same time, instead of having to install them in two separate machines.

Oh, and editors... I suggest using VS Code or pycharm for the editor.

Click to expand...

Thanks for this. So lets say i navigate to the website, I would need to use firefox driver and enter login details ? correct ?

KlaudiaK said:

Thanks for this. So lets say i navigate to the website, I would need to use firefox driver and enter login details ? correct ?

Click to expand...

Chrome driver in my example. But same for FF as well.

Reading the Python or Pip documentation is pretty pointless if you are coming from a fresh background.

I'd recommend finding a course which will cover the basics of programming concepts and then dive further into Python. There's quite a few courses on Udemy or Team Treehouse.

Once you get a little more confident I'd recommend picking a project to work on as this will really help you develop and apply your skills. If you cannot think of a project try to develop something that already exists, e.g. something like Twitter would be ideal.

Nice tidbits in here!