[CODE] People Also Ask Scraping with Python [Upgraded code from iam_ironman]

First of all, thanks for the script you shared.
Can you elaborate what you mean by "hidden apis"? I only know how to call public api, not have clue about hidden apis :(
WhiteHatCoder allready explained, I'll just expand with an example:

Right now, my Selenium script is just imitating user clicks. This is needed, because google loads the elements (answers and additional questions) dynamicly. The additional stuff is not in the original source code, that's why the original script only gives limited questions and no answers.
But this isn't magic of cause, there is for example some JS calling the data from an API, wich is the hidden API. A lot of bigger pages, that seperate back- and frontend often do this.
The fun part is, getting in this API on your own and make these requests just by script. You're just immitating the communication, the Website JS for example does with it's backend, pretending, you're the Website. This is way more lightweigt and so much faster, you can basicly make some dozend to some hundret (I would not recommence that, this can hurt the System you're scraping and we only want data, not ddos) requests per second, while selenium perhaps gets 1-2 per second. Also in most cases you don't have to fight as much against anti-bot measures. You can do this for example by investigation of the JS or with an App that analyzes the Requests, that you make,when you do st.

One could ask, if this is still scraping or hacking, but as long, as it's just calling open data - without diversions - I think it's mostly ok in legal terms. Of course only in a way, where you don't go crazy with requests ;)
 
Nice job, cangrats to you and to the other guys. If you can share the source code for this one and maybe for newer version if you build.

Thanks
Happy to post it
and for the developed app, you can download it from the video discription since i can't post the link here

here is the code :
Code:
from PyQt5 import QtCore, QtGui, QtWidgets
from PyQt5.QtGui import QCursor
from PyQt5.QtWidgets import QMessageBox
import requests
from lxml.html import fromstring
import lxml.html

class Ui_UltraByte(object):
    def worker1(self):
        global keyword
        keyword = self.textEdit.toPlainText()
        self.label_3.setText("Running...")
        self.label_3.setStyleSheet("color: rgb(0, 170, 0);")
        self.label_7.setText("Running...")
        self.label_7.setStyleSheet("color: rgb(0, 170, 0);")
        self.label_5.setText("Running...")
        self.label_5.setStyleSheet("color: rgb(0, 170, 0);")
        self.label_9.setText("Running...")
        self.label_9.setStyleSheet("color: rgb(0, 170, 0);")
    
        self.thread[1] = ThreadClass(parent=None,index=1)
        self.thread[1].start()
        self.thread[1].any_signal.connect(self.start_reporting)
        self.buttontarget.setEnabled(False)
        

    def stopReporting(self):
        self.thread[1].stop()
        self.buttontarget.setEnabled(True)
        self.label_5.setText("Stop...")
        self.label_5.setStyleSheet("color: rgb(225, 0, 0);")
        self.label_9.setText("Stop...")
        self.label_9.setStyleSheet("color: rgb(225, 0, 0);")
    
    def start_reporting(self):
        self.questions.setPlainText(infos)
    
    
        
        
      
    def __init__(self):
        super().__init__() #pour le lancement pour le premier fonction


 
    def openUrl(self):
        url = QtCore.QUrl('https://ultrabyte.agency')
        if not QtGui.QDesktopServices.openUrl(url):
            QtGui.QMessageBox.warning(self, 'Open Url', 'Could not open url')
    
    def setupUi(self, UltraByte):
        self.thread={}
        UltraByte.setObjectName("UltraByte")
        UltraByte.setFixedSize(574, 572)
        UltraByte.setWindowFlags(QtCore.Qt.WindowCloseButtonHint | QtCore.Qt.WindowMinimizeButtonHint)
        icon = QtGui.QIcon()
        icon.addPixmap(QtGui.QPixmap("./res/log.png"), QtGui.QIcon.Normal, QtGui.QIcon.Off)
        UltraByte.setWindowIcon(icon)
        UltraByte.setStyleSheet("background-color: rgb(0, 0, 0);\n"
"color: rgb(255, 255, 255);")
        self.centralwidget = QtWidgets.QWidget(UltraByte)
        self.centralwidget.setObjectName("centralwidget")
        self.label = QtWidgets.QLabel(self.centralwidget)
        self.label.setGeometry(QtCore.QRect(20, 10, 171, 101))
        self.label.setText("")
        self.label.setPixmap(QtGui.QPixmap("./res/log.png"))
        self.label.setObjectName("label")
        self.groupBox = QtWidgets.QGroupBox(self.centralwidget)
        self.groupBox.setGeometry(QtCore.QRect(210, 0, 351, 111))
        self.groupBox.setObjectName("groupBox")
        self.label_2 = QtWidgets.QLabel(self.groupBox)
        self.label_2.setGeometry(QtCore.QRect(20, 20, 71, 21))
        self.label_2.setObjectName("label_2")
        self.label_3 = QtWidgets.QLabel(self.groupBox)
        self.label_3.setGeometry(QtCore.QRect(90, 20, 241, 21))
        self.label_3.setObjectName("label_3")
        self.label_4 = QtWidgets.QLabel(self.groupBox)
        self.label_4.setGeometry(QtCore.QRect(20, 40, 51, 21))
        self.label_4.setObjectName("label_4")
        self.label_5 = QtWidgets.QLabel(self.groupBox)
        self.label_5.setGeometry(QtCore.QRect(80, 40, 251, 21))
        self.label_5.setObjectName("label_5")
        self.label_6 = QtWidgets.QLabel(self.groupBox)
        self.label_6.setGeometry(QtCore.QRect(20, 60, 51, 21))
        self.label_6.setObjectName("label_6")
        self.label_7 = QtWidgets.QLabel(self.groupBox)
        self.label_7.setGeometry(QtCore.QRect(60, 60, 261, 21))
        self.label_7.setObjectName("label_7")
        self.label_8 = QtWidgets.QLabel(self.groupBox)
        self.label_8.setGeometry(QtCore.QRect(20, 80, 51, 21))
        self.label_8.setObjectName("label_8")
        self.label_9 = QtWidgets.QLabel(self.groupBox)
        self.label_9.setGeometry(QtCore.QRect(70, 80, 261, 21))
        self.label_9.setStyleSheet("")
        self.label_9.setObjectName("label_9")
        self.label_10 = QtWidgets.QLabel(self.groupBox)
        self.label_10.setGeometry(QtCore.QRect(140, 110, 51, 21))
        self.label_10.setObjectName("label_10")
        self.groupBox_2 = QtWidgets.QGroupBox(self.centralwidget)
        self.groupBox_2.setGeometry(QtCore.QRect(20, 180, 541, 321))
        self.groupBox_2.setObjectName("groupBox_2")
        self.questions = QtWidgets.QPlainTextEdit(self.groupBox_2)
        self.questions.setGeometry(QtCore.QRect(10, 20, 521, 291))
        self.questions.setObjectName("questions")
        self.buttontarget = QtWidgets.QPushButton(self.centralwidget)
        self.buttontarget.setEnabled(True)
        self.buttontarget.setGeometry(QtCore.QRect(350, 130, 101, 31))
        font = QtGui.QFont()
        font.setFamily("Segoe Print")
        font.setBold(True)
        font.setWeight(75)
        self.buttontarget.setFont(font)
        self.buttontarget.setAcceptDrops(False)
        self.buttontarget.setStyleSheet("QPushButton::chunk{\n"
"    font: 8pt \"MS Shell Dlg 2\";\n"
"    font: 75 8pt \"MS Shell Dlg 2\";\n"
"    border-radius: 10px;\n"
"    background-color: qlineargradient(spread:pad, x1:0, y1:0.511364, x2:1, y2:0.523, stop:0 rgba(254, 121, 199, 255), stop:1 rgba(170, 85, 255, 255));\n"
"}\n"
"\n"
"QPushButton {\n"
"        background-color: rgb(255, 255, 255);\n"
"    color: rgb(0,0,0);\n"
"    border-style: none;\n"
"    border-radius: 10px;\n"
"    text-align: center;\n"
"    height: 50px;\n"
"border: 2px solid black\n"
"}\n"
"\n"
"")
        self.buttontarget.setObjectName("buttontarget")
        self.buttontarget.setCursor(QCursor(QtCore.Qt.PointingHandCursor))
        self.buttontarget_2 = QtWidgets.QPushButton(self.centralwidget)
        self.buttontarget_2.setEnabled(True)
        self.buttontarget_2.setGeometry(QtCore.QRect(460, 130, 101, 31))
        font = QtGui.QFont()
        font.setFamily("Segoe Print")
        font.setBold(True)
        font.setWeight(75)
        self.buttontarget_2.setFont(font)
        self.buttontarget_2.setAcceptDrops(False)
        self.buttontarget_2.setStyleSheet("QPushButton::chunk{\n"
"    font: 8pt \"MS Shell Dlg 2\";\n"
"    font: 75 8pt \"MS Shell Dlg 2\";\n"
"    border-radius: 10px;\n"
"    background-color: qlineargradient(spread:pad, x1:0, y1:0.511364, x2:1, y2:0.523, stop:0 rgba(254, 121, 199, 255), stop:1 rgba(170, 85, 255, 255));\n"
"}\n"
"\n"
"QPushButton {\n"
"    background-color: rgb(255, 255, 255);\n"
"    color: rgb(0,0,0);\n"
"    border-style: none;\n"
"    border-radius: 10px;\n"
"    text-align: center;\n"
"    height: 50px;\n"
"border: 2px solid black\n"
"}\n"
"\n"
"")
        self.buttontarget_2.setObjectName("buttontarget_2")
        self.buttontarget_2.setCursor(QCursor(QtCore.Qt.PointingHandCursor))
        self.textEdit = QtWidgets.QTextEdit(self.centralwidget)
        self.textEdit.setGeometry(QtCore.QRect(120, 130, 211, 31))
        self.textEdit.setObjectName("textEdit")
        self.label_11 = QtWidgets.QLabel(self.centralwidget)
        self.label_11.setGeometry(QtCore.QRect(50, 130, 61, 31))
        self.label_11.setObjectName("label_11")
        self.label_12 = QtWidgets.QLabel(self.centralwidget)
        self.label_12.setGeometry(QtCore.QRect(290, 510, 281, 21))
        self.label_12.setStyleSheet("")
        self.label_12.setObjectName("label_12")
        UltraByte.setCentralWidget(self.centralwidget)
        self.statusbar = QtWidgets.QStatusBar(UltraByte)
        self.statusbar.setObjectName("statusbar")
        UltraByte.setStatusBar(self.statusbar)
        self.menubar = QtWidgets.QMenuBar(UltraByte)
        self.menubar.setGeometry(QtCore.QRect(0, 0, 574, 21))
        self.menubar.setObjectName("menubar")
        self.menuMenu = QtWidgets.QMenu(self.menubar)
        self.menuMenu.setObjectName("menuMenu")
        self.menuHelp_2 = QtWidgets.QMenu(self.menubar)
        self.menuHelp_2.setObjectName("menuHelp_2")
        UltraByte.setMenuBar(self.menubar)
        self.actionClose = QtWidgets.QAction(UltraByte)
        self.actionClose.setObjectName("actionClose")
        self.actionContact_Us = QtWidgets.QAction(UltraByte)
        self.actionContact_Us.setObjectName("actionContact_Us")
        self.menuMenu.addAction(self.actionClose)
        self.menuHelp_2.addAction(self.actionContact_Us)
        self.menubar.addAction(self.menuMenu.menuAction())
        self.menubar.addAction(self.menuHelp_2.menuAction())

        self.retranslateUi(UltraByte)
        QtCore.QMetaObject.connectSlotsByName(UltraByte)

    def retranslateUi(self, UltraByte):
        _translate = QtCore.QCoreApplication.translate
        UltraByte.setWindowTitle(_translate("UltraByte", "PAA GENERATOR"))
        self.groupBox.setTitle(_translate("UltraByte", "System Infos: "))
        self.label_2.setText(_translate("UltraByte", "Server-node : "))
        self.label_3.setText(_translate("UltraByte", "Ready  "))
        self.label_4.setText(_translate("UltraByte", "- Process : "))
        self.label_5.setText(_translate("UltraByte", "Ready  "))
        self.label_6.setText(_translate("UltraByte", "- Sys : "))
        self.label_7.setText(_translate("UltraByte", "Ready  "))
        self.label_8.setText(_translate("UltraByte", "- Script  : "))
        self.label_9.setText(_translate("UltraByte", "Ready  "))
        self.label_10.setText(_translate("UltraByte", "- Script  : "))
        self.groupBox_2.setTitle(_translate("UltraByte", "PAA Generator : "))
        self.buttontarget.setText(_translate("UltraByte", "Launch"))
        self.buttontarget_2.setText(_translate("UltraByte", "Stop"))
        self.label_11.setText(_translate("UltraByte", "KEYWORD  : "))
        self.label_12.setText(_translate("UltraByte", "Powered By xDyablo"))
        self.menuMenu.setTitle(_translate("UltraByte", "File"))
        self.menuHelp_2.setTitle(_translate("UltraByte", "Help"))
        self.actionClose.setText(_translate("UltraByte", "Close"))
        self.actionContact_Us.setText(_translate("UltraByte", "Contact Us"))
        
        self.buttontarget.clicked.connect(self.worker1)
        self.buttontarget_2.clicked.connect(self.stopReporting)
        self.actionClose.triggered.connect(self.openUrl)
        self.actionContact_Us.triggered.connect(self.openUrl)

#Thread Class
# should be  in the run function and the main function
        
#
class ThreadClass(QtCore.QThread):
    any_signal = QtCore.pyqtSignal(str)
    #result of the op1
    def __init__(self, parent=None,index=0):
        super(ThreadClass, self).__init__(parent)
        self.index=index
        self.is_running = True
        
    def TargetEmail(self):
        quotefile = open('./data/Email_Target.txt','r')
        for line in quotefile:
            Target = line.strip()
    
        return Target
        
                
    def run(self):
        global infos
        
        query = keyword
        # main function (paa generator)
        header = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36',
        'Accept-Language': 'en-US,en;q=0.9',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
        }
        
        response = requests.get(f'https://www.google.com/search?q={query}&start=0', headers=header).text
        tree = lxml.html.fromstring(response)
        yourList = tree.xpath('//@data-q')
        butFirst = yourList[1:]
        infos = "\n".join(butFirst)
        self.any_signal.emit(infos)
        
    def stop(self):
        self.is_running = False
        print('Stopping thread...',self.index)
        self.terminate()



if __name__ == "__main__":
    import sys
    app = QtWidgets.QApplication(sys.argv)
    UltraByte = QtWidgets.QMainWindow()
    ui = Ui_UltraByte()
    ui.setupUi(UltraByte)
    UltraByte.show()
    sys.exit(app.exec_())
 
Beeing enthusiastic about st is the first step of learning. And learning is the first step of mastering ;)
Of course, these scripts - mine included, I'm a really bad coder, but I can solve problems quick and dirty xD - are not usefull for any productional purpose. But they solve some nasty starting stuff, like handling xpathes etc. And if people use it, try to understand it and gain their learning it's worth it in the end ;)
It's more about prototyping - and prototypes should be thrown away, after proof of concept. The concept is the point.
as long is do the job , nothing matters : even if it is dirty , you can build a great software with it , keep improving
 
If someone knows python then proxies too can be scraped using the python code
 
If someone knows python then proxies too can be scraped using the python code
if anyone interested here the code to scrape proxies for free :
Python:
def get_proxy():
    url = "https://sslproxies.org"
    r = requests.get(url)
    soup = BeautifulSoup(r.content,'html5lib')
    return {'https' : choice(list(map(lambda x:x[0]+':'+x[1],list(zip(map(lambda x:x.text, soup.findAll('td')[::8]),
                                     map(lambda x:x.text, soup.findAll('td')[1::8]))))))}

but i don't think that free proxies are effective, good luck
 
For any reasons, I can't quote myself, so I'll just copy it =D
I wouldn't call it working, though - it's still rough and buggy, only a prototype reade for rewriting ;)
Python:
from seleniumwire import webdriver
import random
import time
##################################################################
##################################################################
##################################################################
# Your Keyword List
kws = ["webscraping", "bitcoin", "chicken"]

# Limit scraped Questions ||| 0 == All Possible
limit = 15

##################################################################
##################################################################
##################################################################
##### FUNCTIONS
def clickyThingyXpath(xpath, wait, timeout, desc):
    startTime = time.time()
    time.sleep(wait)
    while True:
        checkTime = time.time() - timeout

        if startTime < checkTime:
            return(False)
        try:
            driver.find_element("xpath", xpath).click()
            return(True)
        except:
            time.sleep(1)
            pass

def findTexts(xpath, sleep):
    time.sleep(sleep)
    texts = list()
    _texts = driver.find_elements("xpath",xpath)
    for t in _texts:
        if len(t.text) > 5:
            texts.append(t.text)
    return(texts)

def findQuesionID(questions):
    _id = driver.find_element("xpath", '//*[text() = "'+questions+'"]/parent::*/parent::*/parent::*/parent::*/parent::*').get_attribute("id")
    return(_id)

def XPathByQuestionID(_id):
    xpath = '//*[@id="'+_id+'"]/div/div/div[1]/div[4]'
    return(xpath)

def XPathByAnswerID(_id):
    clicky = '//*[@id="'+_id+'"]//*[@data-attrid="wa:/description"]'
    return(clicky)

def handleAnswerList(_id):
    _dict = dict()

    _dict["heading"] = findTexts('//*[@id="'+_id+'"]//*[@role="heading"]',0)
    findList = findTexts('//*[@id="'+_id+'"]//li',0)
    if len(findList) > 0:
        _dict["list"] = findList
        _dict["anwerType"] = "list"
        return(_dict)
    else:
        return(None)

def doWhateverYouWandwithThe(results):
    import json
    with open("results.json", "w") as jsonFile:
        json.dump(results, jsonFile, indent=4)

##################################################################
##################################################################
##################################################################
##### Script

results = dict()
for kw in kws:
    results[kw] = dict()
    ### Setting Up Webdriver
    ua = ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36","Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36"]
    options = webdriver.ChromeOptions()
    chrome_prefs = {}
    options.experimental_options["prefs"] = chrome_prefs
    chrome_prefs["profile.default_content_settings"] = {"images": 2}
    chrome_prefs["profile.managed_default_content_settings"] = {"images": 2}
    options.add_argument("--disable-web-security")
    options.add_argument("--disable-site-isolation-trials")
    options.add_argument("--disable-application-cache")
    options.add_experimental_option('excludeSwitches', ['enable-logging'])
    options.add_argument('--user-agent='+random.choice(ua))
    driver = webdriver.Chrome(executable_path=r"chromedriver.exe", chrome_options=options)

    ### Request
    url = 'https://www.google.com/search?q='+kw+'&start=0'
    driver.get(url)

    ### Click on Privacy Stuff - Optional
    clickyThingyXpath("/html/body/div[3]/div[3]/span/div/div/div/div[3]/div[1]/button[1]/div", 1, 60, "Handling Privacy Popup")

    ### Scraping
    seen = list()
    questions = findTexts("//*[@data-q]//div[@role='button']//div/span", 0)
    i = 0
    while True:
        for q in questions:
            try:
                # Preparing result dict
                results[kw][str(i)] = {"question":{}, "answer":{}}
                # check if question done allready, since in loop
                if q in seen:
                    continue
                # Get the question ID for further xpath usage
                qid = findQuesionID(q)
                print(kw+" ||| "+q)
                 # Open question
                clickyThingyXpath(XPathByQuestionID(qid), 0, 10, 'Clicking Question: "'+q+'"')
                # Scrape answer
                answer = findTexts(XPathByAnswerID(qid), 2)
                # check if answer is text only
                if len(answer) == 1:
                    anwerType = "Text"
                    while answer[0] == "":
                        answer = findTexts(XPathByAnswerID(qid), 2)
                    # Add to result dict
                    results[kw][str(i)]["question"] = q
                    results[kw][str(i)]["answer"]["answerType"] = "text"
                    results[kw][str(i)]["answer"]["answer"] = answer
                else:
                    # Not Text, so handle as list
                    answer = handleAnswerList(qid)
                    # Ignore if it's not a List as well
                    if answer == None:
                        seen.append(q)
                        continue
                    # Add to result dict
                    results[kw][str(i)]["question"] = q
                    results[kw][str(i)]["answer"] = answer
                seen.append(q)
                i+=1
                if i == limit:
                    break
            except:
                seen.append(q)
                continue
        if i == limit:
            break
        ### Did more questions pop up?
        _questions = findTexts("//*[@data-q]//div[@role='button']//div/span", 0)
        if len(questions) == len(_questions):
            break
        questions = _questions

    driver.close()

doWhateverYouWandwithThe(results)
I'm trying to run the code but I'm getting this error! What am I missing?
I have python and selenium wire installed.
Capture.JPG

Capture2.JPG
 
I'm trying to run the code but I'm getting this error! What am I missing?
I have python and selenium wire installed.
View attachment 226561
View attachment 226562
You can't execute python code in console only - save the script as a .py file in a folder of your choice. copy the folder path. Now go to console and navigate to this folder via cd, pasting the folder name behind it (should look like like cd desktop/yourfoldername). press enter. type in:
python yourfilename.py
you can open python in the console with just typing python as well - but this is only usefull for small stuff.

But you should really first learn the very basics, you will still need to install stuff, download the right webdriver, etc - I can't make tutorials for the lowest levels of python skills, those things you will have to figure out on your own ;)
 
You can't execute python code in console only - save the script as a .py file in a folder of your choice. copy the folder path. Now go to console and navigate to this folder via cd, pasting the folder name behind it (should look like like cd desktop/yourfoldername). press enter. type in:
python yourfilename.py
you can open python in the console with just typing python as well - but this is only usefull for small stuff.

But you should really first learn the very basics, you will still need to install stuff, download the right webdriver, etc - I can't make tutorials for the lowest levels of python skills, those things you will have to figure out on your own ;)
Thanks. Just started learning python, currently at chapter 1 of the "Automate the Boring Stuff with Python". Have much to learn
 
in c
I get this; any ideas about what is not okay?
Code:
Traceback (most recent call last):
  File "paascript.py", line 18, in <module> 
    x = node[0]
IndexError: list index out of range
in case you still need to run the script, if you get that error, it meant your search query returns an empty list.
For example, if you have the keyword "movie" in your keyword.txt, it might return an empty list [] .
so check your keyword list
 
@Bugfisch How can many keywords can we run on the script that you provided here?
It's basicly not limited by the script itself, just by time and proxy costs =D
Since my version is not paralized, one keyword needs a while to finish. If you limit the answers, you might get perhaps 200 per hour, 4800 per day. You will need to integrate a database for better handling, though.
If you parallize it, you can mine more, x20 shouldnt be a problem. But as said - google-friendly proxies or captcha breakers will be needed on some point, so this will become a bit expansive.
As said - the script is more a prototype, not for productive work.
 
It's basicly not limited by the script itself, just by time and proxy costs =D
Since my version is not paralized, one keyword needs a while to finish. If you limit the answers, you might get perhaps 200 per hour, 4800 per day. You will need to integrate a database for better handling, though.
If you parallize it, you can mine more, x20 shouldnt be a problem. But as said - google-friendly proxies or captcha breakers will be needed on some point, so this will become a bit expansive.
As said - the script is more a prototype, not for productive work.
I am already doing 20 keywords with 10-12 answers.

So if i increase the number from 20 to 40 or 50, will I need the proxies?
 
Back
Top