[CODE] People Also Ask Scraping with Python [Upgraded code from iam_ironman]

[Bugfisch]

oh, the code is actually pretty good, especially true if you are just starting out.

Not really, but not professional or st like this. But this was only a 2 hour session based on another script of mine anyway, that I use to generate Bing API codes =D
But it has it flaws - it works though^^

Was referring to other aspects, otherwise hats off to you for doing this.

didn't think you mean the Code Snippets, but the people using it, without really knowing, what they do
But idk - I started like this as well. Better than using just tools, they will learn to read the code on one point, changing it, perhaps they will beginn writing their own.
In the end, we're just throwing out some breadcrumps, no one can build a bakery business on snippets like these. But some might get hungry for bread an start baking.

 

[desperado596]

import requests
import threading
from lxml.html import fromstring
import lxml.html
import time

file1 = open('keywords.txt', 'r')
file2 = open('paa.txt', 'w')
Lines = file1.readlines()
header = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36',
        'Accept-Language': 'tr-tr,en;q=0.9',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
        }

exitFlag = 0
number=0
class myThread (threading.Thread):
   def __init__(self, threadID, name, counter):
      threading.Thread.__init__(self)
      self.threadID = threadID
      self.name = name
      self.counter = counter
   def run(self):     
        print ("Starting " + self.name)
        global number
        query = Lines[number].strip()     
        number = number+1
        response = requests.get(f'https://www.google.com/search?q={query}&start=0', headers=header).text
        tree = lxml.html.fromstring(response)
        node = tree.xpath('//@data-q')
        x = node[0]
        y = node[1]
        z = node[2]
        print("Keyword Processed: " + query)
        file2.write(query)
        file2.write(": ")
        file2.write(x)
        file2.write(" , ")
        file2.write(y)
        file2.write(" , ")
        file2.write(z)
        file2.write("\n")
        print ("Exiting " + self.name)
      

def print_time(threadName, delay, counter):
   while counter:
      if exitFlag:
         threadName.exit()
      time.sleep(delay)
      print ("%s: %s" % (threadName, time.ctime(time.time())))
      counter -= 1

# Create new threads
thread1 = myThread(1, "Thread-1", 1)
thread2 = myThread(2, "Thread-2", 2)

# Start new Threads
thread1.start()
thread2.start()
thread1.join()
thread2.join()

# Closing File
file1.close()
file2.close()

Sure. This is example using 2 multi thread. We can multiply thread just adding thread, or just loop it using for each.

who would want to add proxy function to this ?

 

[desperado596]

Made some upgrades: Multi-threading and proxies support. With this version you can specify how many threads to run at once.

Put these files all in the same folder:

keywords.txt (contains a list of your keywords for google, one per line)

proxies.txt (insert your proxies according to the example below)

http://username:password@ip:port
socks5://username:password@ip:port

Put the below Python code in a file called paa.py in the same folder as the two other files. Make sure you've installed Python on your system. Run it like:



It will output a new file called paa.txt which is the result.

#People Also Ask Multithreaded GoogleYoinker with Proxy Support

import requests
import threading
from lxml.html import fromstring
import lxml.html
from time import time
from queue import Queue
import logging
import random

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)

max_threads = 10 # don't go too crazy
proxy_scheme = 'http' # https, ftp

keywords_list = open('keywords.txt', 'r')
output = open('paa.txt', 'w')

# List of proxies goes in proxies.txt, format one proxy per line:
# http://username:password@ip:port
# socks5://username:password@ip:port
proxies_list = open('proxies.txt', 'r')

keywords = keywords_list.readlines()
proxies = proxies_list.readlines()

if not proxies:
    proxies = None

header = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36',
        'Accept-Language': 'tr-tr,en;q=0.9',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
        }

class GoogleYoinker(threading.Thread):
   def __init__(self, queue):
      threading.Thread.__init__(self)
      self.queue = queue

   def run(self):
        print ("Starting " + self.name)
        while True:
            # Get the work from the queue and expand the tuple
            keyword = self.queue.get()
            try:
                query = keyword.strip()
                if proxies is not None:
                    proxy = {proxy_scheme: proxies[random.randint(0, len(proxies) - 1)].strip()}
                    logger.info("Using Proxy {}".format(proxy))
                    response = requests.get(f'https://www.google.com/search?q={query}&start=0', headers=header, proxies=proxy).text
                else:
                    response = requests.get(f'https://www.google.com/search?q={query}&start=0', headers=header).text
                tree = lxml.html.fromstring(response)
                nodes = tree.xpath('//@data-q')

                logger.info("Keyword Processed: {}".format(query))
                output.write(query)
                output.write(": ")
                for node in nodes:
                    output.write(node)
                    output.write(" , ")
                output.write("\n")

                logger.info('Exiting: {}'.format(self.name))
            finally:
                self.queue.task_done()


def main():
    ts = time()
    queue = Queue()

    # Create up to max_threads worker threads
    for x in range(max_threads):
        worker = GoogleYoinker(queue)

        # Setting daemon to True will let the main thread exit even though the workers are blocking
        worker.daemon = True
        worker.start()

    # Put the keywords into the queue
    for keyword in keywords:
        logger.info('Queueing keyword: {}'.format(keyword))
        queue.put((keyword.strip()))

    # Causes the main thread to wait for the queue to finish processing all the tasks
    queue.join()
    logging.info('Took %s', time() - ts)

    keywords_list.close()
    proxies_list.close()
    output.close()


if __name__ == '__main__':
    main()

thanks for this beautiful code. i tried but my output was really different because of proxy countries. sometimes english sometimes france sometimes turkish output.
Can you do something about it ? thanks
also adding user agent list with random assignment would be good for not getting captcha .

 

[shog]

thanks for this beautiful code. i tried but my output was really different because of proxy countries. sometimes english sometimes france sometimes turkish output.
Can you do something about it ? thanks
also adding user agent list with random assignment would be good for not getting captcha .

There are some old comments about using the "&cr=countryUS" parameter on the search query but some people think it doesn't work any more. It may be far easier to select a proxy provider who can give IPs from the country you need results for.

 

[Thunder Digitals]

I got inspired and created a more advanced version, wich can mostly scrape all questions and it's answers. You will need seleniumwire to run it, though.
Disclaimer: It's still very buggy and dirty coded - but serves as a start. It can also only handle text- and list answers, yet, no tables for example.
Can be used with proxies (thats why seleniumwire, not just selenium) or VPN even , most effective with some IP changing stuff like surfshark and of course be upgraded with captcha breakers or something like this, multithreaded with joblib, etc. You're free to edit it in any way

from seleniumwire import webdriver
from parsel import Selector
import random
import time

##################################################################
##################################################################
##################################################################
# Your Keyword List
kws = ["webscraping", "bitcoin", "chicken"]

# Limit scraped Questions ||| 0 == All Possible
limit = 15

##################################################################
##################################################################
##################################################################
##### FUNCTIONS
def clickyThingyXpath(xpath, wait, timeout, desc):
    startTime = time.time()
    time.sleep(wait)
    while True:
        checkTime = time.time() - timeout

        if startTime < checkTime:
            print("TIMEOUT: clickyThingyXpath["+xpath+"]")
            raise Exception("TIMEOUT: clickyThingyXpath["+xpath+"]")
        try:
            driver.find_element_by_xpath(xpath).click()
            return("SUCCESS: "+desc)
        except:
            time.sleep(1)
            pass
def findTexts(xpath, sleep):
    time.sleep(sleep)
    texts = list()
    _texts = driver.find_elements_by_xpath(xpath)
    for t in _texts:
        if len(t.text) > 5:
            texts.append(t.text)
    return(texts)

def findQuesionID(questions):
    _id = driver.find_element_by_xpath('//*[text() = "'+questions+'"]/parent::*/parent::*/parent::*/parent::*/parent::*').get_attribute("id")
    return(_id)   

def XPathByQuestionID(_id):
    xpath = '//*[@id="'+_id+'"]/div/div/div[1]/div[4]'
    return(xpath)

def XPathByAnswerID(_id):
    clicky = '//*[@id="'+_id+'"]//*[@data-attrid="wa:/description"]'
    return(clicky)

def handleAnswerList(_id):
    _dict = dict()

    _dict["heading"] = findTexts('//*[@id="'+_id+'"]//*[@role="heading"]',0)
    findList = findTexts('//*[@id="'+_id+'"]//li',0)
    if len(findList) > 0:
        _dict["list"] = findList
        _dict["anwerType"] = "list"
        return(_dict)
    else:
        return(None)

def doWhateverYouWandwithThe(results):
    import json
    with open("results.json", "w") as jsonFile:
        json.dump(results, jsonFile, indent=4)

##################################################################
##################################################################
##################################################################
##### Script

results = dict()
for kw in kws:
    results[kw] = dict()
    ### Setting Up Webdriver
    ua = ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36","Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36"]
    options = webdriver.ChromeOptions()
    chrome_prefs = {}
    options.experimental_options["prefs"] = chrome_prefs
    chrome_prefs["profile.default_content_settings"] = {"images": 2}
    chrome_prefs["profile.managed_default_content_settings"] = {"images": 2}
    options.add_argument("--disable-web-security")
    options.add_argument("--disable-site-isolation-trials")
    options.add_argument("--disable-application-cache")
    options.add_experimental_option('excludeSwitches', ['enable-logging'])
    options.add_argument('--user-agent='+random.choice(ua))
    driver = webdriver.Chrome(executable_path=r"chromedriver.exe", chrome_options=options)

    ### Request
    url = 'https://www.google.com/search?q='+kw+'&start=0'
    driver.get(url)

    ### Click on Privacy Stuff - Optional
    # clickyThingyXpath('/html/body/div[3]/div[3]/span/div/div/div/div[3]/div[1]/button[1]/div', 1, 60, "Handling Privacy Popup")

    ### Scraping
    seen = list()
    questions = findTexts("//*[@data-q]//div[@role='button']//div/span", 0)
    i = 0
    while True:
        for q in questions:
            try:
                # Preparing result dict
                results[kw][str(i)] = {"question":{}, "answer":{}}
                # check if question done allready, since in loop
                if q in seen:
                    continue
                # Get the question ID for further xpath usage
                qid = findQuesionID(q)
                print(kw+" ||| "+q)
                 # Open question
                clickyThingyXpath(XPathByQuestionID(qid), 0, 10, 'Clicking Question: "'+q+'"')
                # Scrape answer
                answer = findTexts(XPathByAnswerID(qid), 2)
                # check if answer is text only
                if len(answer) == 1:
                    anwerType = "Text"
                    while answer[0] == "":
                        answer = findTexts(XPathByAnswerID(qid), 2)
                    # Add to result dict
                    results[kw][str(i)]["question"] = q
                    results[kw][str(i)]["answer"]["answerType"] = "text"
                    results[kw][str(i)]["answer"]["answer"] = answer
                else:
                    # Not Text, so handle as list
                    answer = handleAnswerList(qid)
                    # Ignore if it's not a List as well
                    if answer == None:
                        seen.append(q)
                        continue
                    # Add to result dict
                    results[kw][str(i)]["question"] = q
                    results[kw][str(i)]["answer"] = answer
                seen.append(q)
                i+=1
                if i == limit:
                    break
            except:
                seen.append(q)
                continue
        if i == limit:
            break
        ### Did more questions pop up?
        _questions = findTexts("//*[@data-q]//div[@role='button']//div/span", 0)
        if len(questions) == len(_questions):
            break
        questions = _questions

    driver.close()

doWhateverYouWandwithThe(results)

I am getting these errorr

questions = findTexts("//*[@data-q]//div[@role='button']//div/span", 0)
  _texts = driver.find_elements_by_xpath(xpath)
AttributeError: 'Chrome' object has no attribute 'find_elements_by_xpath'

Could you tell me how to solve these?

 

[Floccer]

You may have a newer version of selenium where that find_elements_by_xpath has been deprecated

New syntax:

driver.find_element("xpath", '//*[@id="element"]')

https://stackoverflow.com/questions...object-has-no-attribute-find-element-by-xpath

 

[Thunder Digitals]

You may have a newer version of selenium where that find_elements_by_xpath has been deprecated

New syntax:

driver.find_element("xpath", '//*[@id="element"]')

https://stackoverflow.com/questions...object-has-no-attribute-find-element-by-xpath

yes, i have the version of 4.6.5

 

[Thunder Digitals]

You may have a newer version of selenium where that find_elements_by_xpath has been deprecated

New syntax:

driver.find_element("xpath", '//*[@id="element"]')

https://stackoverflow.com/questions...object-has-no-attribute-find-element-by-xpath

Could you please update the code for the newer version of selenium wire?

Because I am little difficulty here.

Thank you so much.

 

[Captain bd]

How do i scrap answers?

 

[Bugfisch]

Could you please update the code for the newer version of selenium wire?

Because I am little difficulty here.

Thank you so much.

Should work with the actual version now. Uncomment line 97 if a European Cookie PopUp blocks the screen

from seleniumwire import webdriver
import random
import time
##################################################################
##################################################################
##################################################################
# Your Keyword List
kws = ["webscraping", "bitcoin", "chicken"]

# Limit scraped Questions ||| 0 == All Possible
limit = 15

##################################################################
##################################################################
##################################################################
##### FUNCTIONS
def clickyThingyXpath(xpath, wait, timeout, desc):
    startTime = time.time()
    time.sleep(wait)
    while True:
        checkTime = time.time() - timeout

        if startTime < checkTime:
            return(False)
        try:
            driver.find_element("xpath", xpath).click()
            return(True)
        except:
            time.sleep(1)
            pass

def findTexts(xpath, sleep):
    time.sleep(sleep)
    texts = list()
    _texts = driver.find_elements("xpath",xpath)
    for t in _texts:
        if len(t.text) > 5:
            texts.append(t.text)
    return(texts)

def findQuesionID(questions):
    _id = driver.find_element("xpath", '//*[text() = "'+questions+'"]/parent::*/parent::*/parent::*/parent::*/parent::*').get_attribute("id")
    return(_id)   

def XPathByQuestionID(_id):
    xpath = '//*[@id="'+_id+'"]/div/div/div[1]/div[4]'
    return(xpath)

def XPathByAnswerID(_id):
    clicky = '//*[@id="'+_id+'"]//*[@data-attrid="wa:/description"]'
    return(clicky)

def handleAnswerList(_id):
    _dict = dict()

    _dict["heading"] = findTexts('//*[@id="'+_id+'"]//*[@role="heading"]',0)
    findList = findTexts('//*[@id="'+_id+'"]//li',0)
    if len(findList) > 0:
        _dict["list"] = findList
        _dict["anwerType"] = "list"
        return(_dict)
    else:
        return(None)

def doWhateverYouWandwithThe(results):
    import json
    with open("results.json", "w") as jsonFile:
        json.dump(results, jsonFile, indent=4)

##################################################################
##################################################################
##################################################################
##### Script

results = dict()
for kw in kws:
    results[kw] = dict()
    ### Setting Up Webdriver
    ua = ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36","Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36"]
    options = webdriver.ChromeOptions()
    chrome_prefs = {}
    options.experimental_options["prefs"] = chrome_prefs
    chrome_prefs["profile.default_content_settings"] = {"images": 2}
    chrome_prefs["profile.managed_default_content_settings"] = {"images": 2}
    options.add_argument("--disable-web-security")
    options.add_argument("--disable-site-isolation-trials")
    options.add_argument("--disable-application-cache")
    options.add_experimental_option('excludeSwitches', ['enable-logging'])
    options.add_argument('--user-agent='+random.choice(ua))
    driver = webdriver.Chrome(executable_path=r"chromedriver.exe", chrome_options=options)

    ### Request
    url = 'https://www.google.com/search?q='+kw+'&start=0'
    driver.get(url)

    ### Click on Privacy Stuff - Optional
    # clickyThingyXpath("/html/body/div[3]/div[3]/span/div/div/div/div[3]/div[1]/button[1]/div", 1, 60, "Handling Privacy Popup")

    ### Scraping
    seen = list()
    questions = findTexts("//*[@data-q]//div[@role='button']//div/span", 0)
    i = 0
    while True:
        for q in questions:
            try:
                # Preparing result dict
                results[kw][str(i)] = {"question":{}, "answer":{}}
                # check if question done allready, since in loop
                if q in seen:
                    continue
                # Get the question ID for further xpath usage
                qid = findQuesionID(q)
                print(kw+" ||| "+q)
                 # Open question
                clickyThingyXpath(XPathByQuestionID(qid), 0, 10, 'Clicking Question: "'+q+'"')
                # Scrape answer
                answer = findTexts(XPathByAnswerID(qid), 2)
                # check if answer is text only
                if len(answer) == 1:
                    anwerType = "Text"
                    while answer[0] == "":
                        answer = findTexts(XPathByAnswerID(qid), 2)
                    # Add to result dict
                    results[kw][str(i)]["question"] = q
                    results[kw][str(i)]["answer"]["answerType"] = "text"
                    results[kw][str(i)]["answer"]["answer"] = answer
                else:
                    # Not Text, so handle as list
                    answer = handleAnswerList(qid)
                    # Ignore if it's not a List as well
                    if answer == None:
                        seen.append(q)
                        continue
                    # Add to result dict
                    results[kw][str(i)]["question"] = q
                    results[kw][str(i)]["answer"] = answer
                seen.append(q)
                i+=1
                if i == limit:
                    break
            except:
                seen.append(q)
                continue
        if i == limit:
            break
        ### Did more questions pop up?
        _questions = findTexts("//*[@data-q]//div[@role='button']//div/span", 0)
        if len(questions) == len(_questions):
            break
        questions = _questions

    driver.close()

doWhateverYouWandwithThe(results)

 

[Thunder Digitals]

Should work with the actual version now. Uncomment line 97 if a European Cookie PopUp blocks the screen

from seleniumwire import webdriver
import random
import time
##################################################################
##################################################################
##################################################################
# Your Keyword List
kws = ["webscraping", "bitcoin", "chicken"]

# Limit scraped Questions ||| 0 == All Possible
limit = 15

##################################################################
##################################################################
##################################################################
##### FUNCTIONS
def clickyThingyXpath(xpath, wait, timeout, desc):
    startTime = time.time()
    time.sleep(wait)
    while True:
        checkTime = time.time() - timeout

        if startTime < checkTime:
            return(False)
        try:
            driver.find_element("xpath", xpath).click()
            return(True)
        except:
            time.sleep(1)
            pass

def findTexts(xpath, sleep):
    time.sleep(sleep)
    texts = list()
    _texts = driver.find_elements("xpath",xpath)
    for t in _texts:
        if len(t.text) > 5:
            texts.append(t.text)
    return(texts)

def findQuesionID(questions):
    _id = driver.find_element("xpath", '//*[text() = "'+questions+'"]/parent::*/parent::*/parent::*/parent::*/parent::*').get_attribute("id")
    return(_id)   

def XPathByQuestionID(_id):
    xpath = '//*[@id="'+_id+'"]/div/div/div[1]/div[4]'
    return(xpath)

def XPathByAnswerID(_id):
    clicky = '//*[@id="'+_id+'"]//*[@data-attrid="wa:/description"]'
    return(clicky)

def handleAnswerList(_id):
    _dict = dict()

    _dict["heading"] = findTexts('//*[@id="'+_id+'"]//*[@role="heading"]',0)
    findList = findTexts('//*[@id="'+_id+'"]//li',0)
    if len(findList) > 0:
        _dict["list"] = findList
        _dict["anwerType"] = "list"
        return(_dict)
    else:
        return(None)

def doWhateverYouWandwithThe(results):
    import json
    with open("results.json", "w") as jsonFile:
        json.dump(results, jsonFile, indent=4)

##################################################################
##################################################################
##################################################################
##### Script

results = dict()
for kw in kws:
    results[kw] = dict()
    ### Setting Up Webdriver
    ua = ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36","Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36"]
    options = webdriver.ChromeOptions()
    chrome_prefs = {}
    options.experimental_options["prefs"] = chrome_prefs
    chrome_prefs["profile.default_content_settings"] = {"images": 2}
    chrome_prefs["profile.managed_default_content_settings"] = {"images": 2}
    options.add_argument("--disable-web-security")
    options.add_argument("--disable-site-isolation-trials")
    options.add_argument("--disable-application-cache")
    options.add_experimental_option('excludeSwitches', ['enable-logging'])
    options.add_argument('--user-agent='+random.choice(ua))
    driver = webdriver.Chrome(executable_path=r"chromedriver.exe", chrome_options=options)

    ### Request
    url = 'https://www.google.com/search?q='+kw+'&start=0'
    driver.get(url)

    ### Click on Privacy Stuff - Optional
    # clickyThingyXpath("/html/body/div[3]/div[3]/span/div/div/div/div[3]/div[1]/button[1]/div", 1, 60, "Handling Privacy Popup")

    ### Scraping
    seen = list()
    questions = findTexts("//*[@data-q]//div[@role='button']//div/span", 0)
    i = 0
    while True:
        for q in questions:
            try:
                # Preparing result dict
                results[kw][str(i)] = {"question":{}, "answer":{}}
                # check if question done allready, since in loop
                if q in seen:
                    continue
                # Get the question ID for further xpath usage
                qid = findQuesionID(q)
                print(kw+" ||| "+q)
                 # Open question
                clickyThingyXpath(XPathByQuestionID(qid), 0, 10, 'Clicking Question: "'+q+'"')
                # Scrape answer
                answer = findTexts(XPathByAnswerID(qid), 2)
                # check if answer is text only
                if len(answer) == 1:
                    anwerType = "Text"
                    while answer[0] == "":
                        answer = findTexts(XPathByAnswerID(qid), 2)
                    # Add to result dict
                    results[kw][str(i)]["question"] = q
                    results[kw][str(i)]["answer"]["answerType"] = "text"
                    results[kw][str(i)]["answer"]["answer"] = answer
                else:
                    # Not Text, so handle as list
                    answer = handleAnswerList(qid)
                    # Ignore if it's not a List as well
                    if answer == None:
                        seen.append(q)
                        continue
                    # Add to result dict
                    results[kw][str(i)]["question"] = q
                    results[kw][str(i)]["answer"] = answer
                seen.append(q)
                i+=1
                if i == limit:
                    break
            except:
                seen.append(q)
                continue
        if i == limit:
            break
        ### Did more questions pop up?
        _questions = findTexts("//*[@data-q]//div[@role='button']//div/span", 0)
        if len(questions) == len(_questions):
            break
        questions = _questions

    driver.close()

doWhateverYouWandwithThe(results)

It worked. Thank you so much.

but it only scraping questions.

how can I scrape answers too?

 

[Floccer]

It worked. Thank you so much.

but it only scraping questions.

how can I scrape answers too?

That should save questions and answers to results.json -file on the same folder where you run it. It just won't display them on the prompt

 

[Bugfisch]

how can I scrape answers too?

It's scraping the answers, it's just not printing em in console - most of em, by now only lists and text answers, no tables or videos. You find em in the results.json or edit the doWhateverYouWandwithThe(results) function and save em in a csv, a db or whatever instead.

{
    "webscraping": {
        "0": {
            "question": "Was ist ein Scraping Tool?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Ein Web Scraping Tool kann verschiedene Arten an Informationen extrahieren und diese wiederum in verschiedenen Dateitypen ausgeben. So k\u00f6nnen neben reinem Text beispielsweise auch Bilder und Videos oder E-Mail Adressen und URL's extrahiert werden."
                ]
            }
        },
        "1": {
            "question": "Wie funktioniert ein Web Scraper?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Beim Web Scraping (engl. scraping = \u201ekratzen/absch\u00fcrfen\u201c) werden Daten von Webseiten extrahiert und gespeichert, um diese zu analysieren oder anderweitig zu verwerten. Beim Scraping werden viele verschiedene Arten von Informationen gesammelt."
                ]
            }
        },
        "2": {
            "question": "Ist Crawlen verboten?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Die Rechtslage l\u00e4sst sich so zusammenfassen, dass Screescraping und Webscraping grunds\u00e4tzlich zul\u00e4ssig ist, wenn von den eingesetzten Bots (Crawler, Scraper) keine technischen Schutzma\u00dfnahmen \u00fcberwunden werden und man auch keine eigene \"Schattendatenbank\" mit den Daten best\u00fcckt.01.10.2020"
                ]
            }
        },
        "3": {
            "question": "Ist Web Scraping legal Schweiz?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Zum urheberrechtlichen Schutz der Listen: Datenkompilation ist trotz Aufwand und Wert grunds\u00e4tzlich nicht gesch\u00fctzt, sofern weder die Gestaltung noch die Auswahlkriterien originell sind. Manuelles Eintippen von Daten ist kein technisches Reproduktionsverfahren i.S.v. Art."
                ]
            }
        },
        "4": {
            "question": "Ist ein Impressum in der Schweiz Pflicht?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "In der Schweiz besteht bislang keine generelle Impressumspflicht f\u00fcr Websites. Ein Impressum ist deshalb nur in Ausnahmef\u00e4llen rechtlich notwendig. Nun wird aber im Fr\u00fchling 2012 eine generelle Impressumspflicht f\u00fcr den \u00abelektronischen Gesch\u00e4ftsverkehr\u00bb in der Schweiz eingef\u00fchrt.01.12.2011"
                ]
            }
        },
        "5": {
            "question": "Was muss alles auf einer Website stehen Schweiz?",
            "answer": {
                "heading": [
                    "Es sind die folgenden Angaben zu machen:"
                ],
                "list": [
                    "Vorname/Name bei nat\u00fcrlichen Personen bzw. vollst\u00e4ndige Firma bei Unternehmen.",
                    "Postadresse (Wohnsitz bzw. Sitz und weitere Adressangaben, Postfachnummer gen\u00fcgt nicht)",
                    "E-Mail-Adresse.",
                    "Optional: Telefonnummer sowie Faxnummer (sofern vorhanden)"
                ],
                "anwerType": "list"
            }
        },
        "6": {
            "question": "Was muss in der Schweiz in ein Impressum?",
            "answer": {
                "heading": [
                    "Impressumspflicht in der Schweiz"
                ],
                "list": [
                    "Ihr kompletter Name oder der Name des Unternehmens.",
                    "Ihre Adresse oder der Sitz des Unternehmens (Postfach reicht nicht aus)",
                    "E-Mail-Adresse (Kontaktformular reicht nicht aus. Ausserdem darf die E-Mail-Adresse nicht als Bild \u201egetarnt\u201c werden.)"
                ],
                "anwerType": "list"
            }
        },
        "7": {
            "question": "Was passiert wenn man kein Impressum hat?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Fast jede Webseite im Internet ben\u00f6tigt eine Anbieterkennzeichnung auch Impressum genannt. Im Impressum muss eine aktuelle Adresse angegeben werden, das Impressum muss leicht erreichbar sein. Ist kein Impressum auf einer Webseite vorhanden kann das zu teuren Abmahnungen f\u00fchren.13.07.2022"
                ]
            }
        },
        "8": {
            "question": "Was darf ich auf meiner Homepage ver\u00f6ffentlichen?",
            "answer": {
                "heading": [
                    "Man muss also f\u00fcr alle urheberrechtlich gesch\u00fctzten Werke, die auf der Website erscheinen, das Recht haben, sie zu ver\u00f6ffentlichen."
                ],
                "list": [
                    "Freie Lizenzen. Es ist erlaubt, Inhalte zu verwenden, die vom Urheber ausdr\u00fccklich zur Verwendung freigegeben sind. ...",
                    "Fotos. ...",
                    "Videos. ...",
                    "Musik. ...",
                    "Grafiken. ...",
                    "Texte. ...",
                    "Spezialfall Sch\u00fclerzeitung?"
                ],
                "anwerType": "list"
            }
        },
        "9": {
            "question": "Wer ist verantwortlich f\u00fcr eine Website?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "F\u00fcr eine Website gibt es prinzipiell immer zwei verantwortliche Stelle mit gr\u00f6\u00dftenteils disjunkten Aufgabenbereichen. Dem Betreiber obliegt die Hoheit \u00fcber den eigentlichen Inhalt der Website, also der ver\u00f6ffentlichten Informationen, der Webhoster ist f\u00fcr die (technische) Verf\u00fcgbarkeit der Website verantwortlich.29.04.2016"
                ]
            }
        },
        "10": {
            "question": "Sind AGB auf Website Pflicht?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Grunds\u00e4tzlich gibt es keine Pflicht, AGB auf einer Webseite zu verwenden. Die Nutzung der AGB liegt in der Regel allein im Interesse des Verwenders.24.11.2016"
                ]
            }
        },
        "11": {
            "question": "Ist ein Impressum auf einer Homepage Pflicht?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Die Impressumspflicht gilt f\u00fcr elektronische Medien, die wenigstens viermal im Kalenderjahr in vergleichbarer Gestaltung verbreitet werden, z.B. elektronische Newsletter. Umgangssprachlich werden die Offenlegungspflichten nach dem Mediengesetz h\u00e4ufig als Impressum bezeichnet!01.01.2022"
                ]
            }
        },
        "12": {
            "question": "Wann ist ein Impressum n\u00f6tig?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Jeder, der Telemedien gesch\u00e4ftsm\u00e4\u00dfig oder journalistisch-redaktionell nutzt, ben\u00f6tigt ein Impressum. Dazu z\u00e4hlen jedoch nicht nur Unternehmen, Beh\u00f6rden, Stiftungen, Vereine oder Selbstst\u00e4ndige. Auch bei einem \u00f6ffentlichen Blog oder einer Webseite, die sich durch Werbeeinnahmen finanziert, braucht man ein Impressum.16.10.2020"
                ]
            }
        },
        "13": {
            "question": "Was ist Pflicht bei einer Website?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "In Deutschland ist die Pflicht zu einem Impressum bzw. einer Anbieterkennzeichnung auf \u00a7 5 des Telemediengesetzes (TMG) und \u00a7 18 des Medienstaatsvertrags (MStV) zur\u00fcckzuf\u00fchren. Diese Paragrafen sollen sicherstellen, dass Nutzern zumindest grundlegende Informationen zum Website-Betreiber verf\u00fcgbar sind."
                ]
            }
        },
        "14": {
            "question": "Was muss rechtlich auf eine Website?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Welche Angaben das Impressum beinhalten muss, hat der Gesetzgeber \u00fcberwiegend in \u00a7 5 TMG festgelegt. So muss die Identit\u00e4t des Website-Betreibers preisgegeben werden, also bei einer juristischen Person etwa die Firma, die Rechtsform, die Anschrift und der vollst\u00e4ndige Name des/der Vertretungsberechtigten.03.12.2014"
                ]
            }
        }
    },
    "bitcoin": {
        "0": {
            "question": "How long does it take to mine 1 Bitcoin?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "The average time for generating one Bitcoin is about 10 minutes, but this applies only to powerful machines. The speed of mining depends on the type of Bitcoin mining hardware you are using."
                ]
            }
        },
        "1": {
            "question": "How does Bitcoin make money?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Key Takeaways. By mining, you can earn cryptocurrency without having to put down money for it. Bitcoin miners receive bitcoin as a reward for completing \"blocks\" of verified transactions, which are added to the blockchain."
                ]
            }
        },
        "2": {
            "question": "How do beginners buy bitcoins?",
            "answer": {
                "heading": [
                    "Follow these five steps to buy Bitcoin and keep it secure:"
                ],
                "list": [
                    "Choose a Bitcoin Exchange. Buying Bitcoin requires exchanging your dollars for some of those sweet, sweet golden coins. ...",
                    "Create Your Exchange Account. ...",
                    "Fund Your Exchange Account. ...",
                    "Order Your Bitcoin. ...",
                    "Store Your Bitcoin."
                ],
                "anwerType": "list"
            }
        },
        "3": {
            "question": "Who is Bitcoin owned by?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "who controls Bitcoin? Bitcoin is controlled by all Bitcoin users around the world. Developers are improving the software but they can't force a change in the rules of the Bitcoin protocol because all users are free to choose what software they use."
                ]
            }
        },
        "4": {
            "question": "What is the minimum amount to invest in Bitcoin?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Search for Bitcoin \u2013 Type 'BTC' or 'Bitcoin' into the search bar at the top of your screen and click 'Trade'. Invest \u2013 Enter the amount you'd like to invest in Bitcoin (minimum of $10), double-check everything is correct, and click 'Open Trade'.12.03.2022"
                ]
            }
        },
        "5": {
            "question": "How many bitcoins are left?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "How Many Bitcoins are Left to Mine? How many of the 21 million Bitcoins are left? There are 2.3 million Bitcoin left to be mined. Surprisingly, even though 18.6 million Bitcoin were mined in just over 10 years, it will take another 120 years to mine the remaining 2.3 million."
                ]
            }
        },
        "6": {
            "question": "How can I get 1 Bitcoin for free?",
            "answer": {
                "heading": [
                    "Methods To Earn Free Bitcoins"
                ],
                "list": [
                    "#1) Pionex \u2013 Using Crypto Trading Bots.",
                    "#2) Bitstamp \u2013 Using Staking Rewards.",
                    "#3) Tipping Bots And Platforms.",
                    "#4) Playing Online and Offline Games.",
                    "#5) Mining Browsers And Free Mining Software.",
                    "#6) Earning Free Bitcoins Through Bounties.",
                    "#7) Earn From Crypto Airdrops."
                ],
                "anwerType": "list"
            }
        },
        "7": {
            "question": "How much do bitcoin miners make a day?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Bitcoin mining is still extremely profitable in 2022. Bitcoin miners are currently mining around $20 million worth of Bitcoin per day. That's $600 million per month. A mining machine costs $2,000-$20,000, making it difficult for anyone but professional miners to mine.03.08.2022"
                ]
            }
        },
        "8": {
            "question": "Is it worth buying 100 dollars of Bitcoin?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Ultimately, it's up to you whether investing $100 in Bitcoin is worth it or not. If it's a one-time investment and you just want to try crypto out, we would recommend going with a lower amount since you can't profit much from $100 anyway.30.05.2022"
                ]
            }
        },
        "9": {
            "question": "Can Bitcoin be converted to cash?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "You can use a crypto exchange like Coinbase, Binance, Gemini or Kraken to turn Bitcoin into cash. This may be an easy method if you already use a centralized exchange and your crypto lives in a custodial wallet. Choose the coin and amount you'd like to sell, agree to the rates and your cash will be available to you.30.05.2022"
                ]
            }
        },
        "10": {
            "question": "Can I invest $5 in Bitcoin?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "New users get a $5 credit to invest in Bitcoin via the Bitcoin Investment Trust (GBTC), or any of the 1000+ stocks and ETFs we offer."
                ]
            }
        },
        "11": {
            "question": "How can I sell bitcoins for cash?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "To sell your bitcoin, you just need to go to an exchange and click to sell your bitcoin. Then you can choose how much you want to sell and move your cash to a bank. If you want to turn it straight to cash instead, then you should go through a bitcoin ATM.11.07.2022"
                ]
            }
        },
        "12": {
            "question": "Who gets the money when you buy Bitcoin?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "A buyer and seller agree on a price and a trade is executed over an exchange. So our $50k investor buys that amount of bitcoins and the seller receives the $50k in the form of a cash deposit. That seller may now keep it in the bank, buy other cryptos or withdraw it and spend it in any way they choose.27.11.2017"
                ]
            }
        },
        "13": {
            "question": "Is Bitcoin a good investment 2022?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Bitcoin's price has continued to fall through the first half of 2022, as has the stock market. Investors have turned to assets that perform well in a time of a slowing economy, higher inflation and rising interest rates.22.07.2022"
                ]
            }
        },
        "14": {
            "question": "Is Bitcoin still a good investment 2022?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Experts remain conflicted on whether bitcoin has bottomed out yet. Some say it already has, while others says bitcoin could fall as low as $10,000 in 2022. This volatility is a big part of why experts recommend keeping your crypto investments to less than 5% of your portfolio to begin with.22.07.2022"
                ]
            }
        }
    },
    "chicken": {
        "0": {
            "question": "Is chicken a dinosaur?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "So, are chickens dinosaurs? No \u2013 the birds are a distinct group of animals, but they did descend from the dinosaurs, and it's not too much of a twist of facts to call them modern dinosaurs. There are many similarities between the two types of animal, largely to do with bone structure."
                ]
            }
        },
        "1": {
            "question": "Is chicken a real animal?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "That seems at least partly true, but new research from Uppsala University now shows that the wild origins of the chicken are more complicated. The researchers mapped the genes that give most domesticated chickens yellow legs and found the genetic heredity derives from a closely related species, the gray jungle fowl.29.02.2008"
                ]
            }
        },
        "2": {
            "question": "What are the 3 types of chicken?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Chickens. Many different breeds of chickens have been developed for different purposes. For simplicity, you can place them into three general categories: Laying, meat-producing and dual-purpose breeds."
                ]
            }
        },
        "3": {
            "question": "What is small chicken called?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Miniature, or \u201cBantam,\u201d chickens are a subspecies of chicken that are much smaller than regular chickens. The average chicken weighs 5 to 10 pounds, whereas bantams only weigh 1 to 2.5 pounds! Most chicken breeds have a Bantam or miniature counterpart."
                ]
            }
        },
        "4": {
            "question": "Why are male chickens killed?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Male chicks are killed for two reasons: they cannot lay eggs and they are not suitable for chicken-meat production. This is because layer hens \u2014 and therefore their chicks \u2014 are a different breed of poultry to chickens that are bred and raised for meat production.22.09.2021"
                ]
            }
        },
        "5": {
            "question": "How big is a chicken brain?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "A chicken's brain is relatively small given the size of the head. It is the size of about 2 shelled peanuts, or a little bigger than a chicken's eyeball."
                ]
            }
        },
        "6": {
            "question": "Can chickens change gender?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "It is possible for a female chicken to take on external characteristics of a male, a phenomenon referred to as sex reversal."
                ]
            }
        },
        "7": {
            "question": "Is a hen a girl or boy?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Just as we humans are a particular species within mammals. And just as humans are divided into male and female and children and adults, chickens are likewise. In this context, a man equals a \u201crooster\u201d and a woman equals a \u201chen\u201d. Both are generally chickens, but the names indicate their gender and that they are adults.26.04.2019"
                ]
            }
        },
        "8": {
            "question": "Are there male chickens?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "The answer is yes, all male chickens are roosters. Roosters present a challenge to many backyard chicken owners because local jurisdictions usually have ordinances against roosters because they love to crow, which disturbs the neighbors.02.08.2022"
                ]
            }
        },
        "9": {
            "question": "Why are chickens called chooks?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "A domestic fowl; a chicken. Chook comes from British dialect chuck(y) 'a chicken; a fowl' which is a variant of chick. Chook is the common term for the live bird, although chook raffles, held in Australian clubs and pubs, have ready-to-cook chooks as prizes."
                ]
            }
        },
        "10": {
            "question": "What is a neutered chicken called?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "A capon is a male chicken that is gelded, or castrated, at a young age, and then fed a rich diet of milk or porridge. Larger than a chicken, a bit smaller than a turkey, but more flavorful than either, capons are full breasted with tender, juicy, flavorful meat that is well suited to roasting."
                ]
            }
        },
        "11": {
            "question": "Do dinosaurs Still Exist?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Other than birds, however, there is no scientific evidence that any dinosaurs, such as Tyrannosaurus, Velociraptor, Apatosaurus, Stegosaurus, or Triceratops, are still alive. These, and all other non-avian dinosaurs became extinct at least 65 million years ago at the end of the Cretaceous Period."
                ]
            }
        },
        "12": {
            "question": "What came first chicken or egg?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Back to our original question: with amniotic eggs showing up roughly 340 million or so years ago, and the first chickens evolving at around 58 thousand years ago at the earliest, it's a safe bet to say the egg came first. Eggs were around way before chickens even existed.29.03.2018"
                ]
            }
        },
        "13": {
            "question": "What was the first bird to fly?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Archaeopteryx: The Transitional Fossil. Paleontologists view Archaeopteryx as a transitional fossil between dinosaurs and modern birds. With its blend of avian and reptilian features, it was long viewed as the earliest known bird.14.03.2018"
                ]
            }
        },
        "14": {
            "question": "Can a chicken fly?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Chickens may have wings and fluffy feathers, but they're fairly dismal fliers, often going airborne for only a few yards before landing. The reason for their poor flight isn't as rhetorical as why they crossed the road.08.12.2016"
                ]
            }
        }
    }
}

 

[Thunder Digitals]

That should save questions and answers to results.json -file on the same folder where you run it. It just won't display them on the prompt

It's scraping the answers, it's just not printing em in console - most of em, by now only lists and text answers, no tables or videos. You find em in the results.json or edit the doWhateverYouWandwithThe(results) function and save em in a csv, a db or whatever instead.

{
    "webscraping": {
        "0": {
            "question": "Was ist ein Scraping Tool?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Ein Web Scraping Tool kann verschiedene Arten an Informationen extrahieren und diese wiederum in verschiedenen Dateitypen ausgeben. So k\u00f6nnen neben reinem Text beispielsweise auch Bilder und Videos oder E-Mail Adressen und URL's extrahiert werden."
                ]
            }
        },
        "1": {
            "question": "Wie funktioniert ein Web Scraper?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Beim Web Scraping (engl. scraping = \u201ekratzen/absch\u00fcrfen\u201c) werden Daten von Webseiten extrahiert und gespeichert, um diese zu analysieren oder anderweitig zu verwerten. Beim Scraping werden viele verschiedene Arten von Informationen gesammelt."
                ]
            }
        },
        "2": {
            "question": "Ist Crawlen verboten?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Die Rechtslage l\u00e4sst sich so zusammenfassen, dass Screescraping und Webscraping grunds\u00e4tzlich zul\u00e4ssig ist, wenn von den eingesetzten Bots (Crawler, Scraper) keine technischen Schutzma\u00dfnahmen \u00fcberwunden werden und man auch keine eigene \"Schattendatenbank\" mit den Daten best\u00fcckt.01.10.2020"
                ]
            }
        },
        "3": {
            "question": "Ist Web Scraping legal Schweiz?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Zum urheberrechtlichen Schutz der Listen: Datenkompilation ist trotz Aufwand und Wert grunds\u00e4tzlich nicht gesch\u00fctzt, sofern weder die Gestaltung noch die Auswahlkriterien originell sind. Manuelles Eintippen von Daten ist kein technisches Reproduktionsverfahren i.S.v. Art."
                ]
            }
        },
        "4": {
            "question": "Ist ein Impressum in der Schweiz Pflicht?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "In der Schweiz besteht bislang keine generelle Impressumspflicht f\u00fcr Websites. Ein Impressum ist deshalb nur in Ausnahmef\u00e4llen rechtlich notwendig. Nun wird aber im Fr\u00fchling 2012 eine generelle Impressumspflicht f\u00fcr den \u00abelektronischen Gesch\u00e4ftsverkehr\u00bb in der Schweiz eingef\u00fchrt.01.12.2011"
                ]
            }
        },
        "5": {
            "question": "Was muss alles auf einer Website stehen Schweiz?",
            "answer": {
                "heading": [
                    "Es sind die folgenden Angaben zu machen:"
                ],
                "list": [
                    "Vorname/Name bei nat\u00fcrlichen Personen bzw. vollst\u00e4ndige Firma bei Unternehmen.",
                    "Postadresse (Wohnsitz bzw. Sitz und weitere Adressangaben, Postfachnummer gen\u00fcgt nicht)",
                    "E-Mail-Adresse.",
                    "Optional: Telefonnummer sowie Faxnummer (sofern vorhanden)"
                ],
                "anwerType": "list"
            }
        },
        "6": {
            "question": "Was muss in der Schweiz in ein Impressum?",
            "answer": {
                "heading": [
                    "Impressumspflicht in der Schweiz"
                ],
                "list": [
                    "Ihr kompletter Name oder der Name des Unternehmens.",
                    "Ihre Adresse oder der Sitz des Unternehmens (Postfach reicht nicht aus)",
                    "E-Mail-Adresse (Kontaktformular reicht nicht aus. Ausserdem darf die E-Mail-Adresse nicht als Bild \u201egetarnt\u201c werden.)"
                ],
                "anwerType": "list"
            }
        },
        "7": {
            "question": "Was passiert wenn man kein Impressum hat?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Fast jede Webseite im Internet ben\u00f6tigt eine Anbieterkennzeichnung auch Impressum genannt. Im Impressum muss eine aktuelle Adresse angegeben werden, das Impressum muss leicht erreichbar sein. Ist kein Impressum auf einer Webseite vorhanden kann das zu teuren Abmahnungen f\u00fchren.13.07.2022"
                ]
            }
        },
        "8": {
            "question": "Was darf ich auf meiner Homepage ver\u00f6ffentlichen?",
            "answer": {
                "heading": [
                    "Man muss also f\u00fcr alle urheberrechtlich gesch\u00fctzten Werke, die auf der Website erscheinen, das Recht haben, sie zu ver\u00f6ffentlichen."
                ],
                "list": [
                    "Freie Lizenzen. Es ist erlaubt, Inhalte zu verwenden, die vom Urheber ausdr\u00fccklich zur Verwendung freigegeben sind. ...",
                    "Fotos. ...",
                    "Videos. ...",
                    "Musik. ...",
                    "Grafiken. ...",
                    "Texte. ...",
                    "Spezialfall Sch\u00fclerzeitung?"
                ],
                "anwerType": "list"
            }
        },
        "9": {
            "question": "Wer ist verantwortlich f\u00fcr eine Website?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "F\u00fcr eine Website gibt es prinzipiell immer zwei verantwortliche Stelle mit gr\u00f6\u00dftenteils disjunkten Aufgabenbereichen. Dem Betreiber obliegt die Hoheit \u00fcber den eigentlichen Inhalt der Website, also der ver\u00f6ffentlichten Informationen, der Webhoster ist f\u00fcr die (technische) Verf\u00fcgbarkeit der Website verantwortlich.29.04.2016"
                ]
            }
        },
        "10": {
            "question": "Sind AGB auf Website Pflicht?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Grunds\u00e4tzlich gibt es keine Pflicht, AGB auf einer Webseite zu verwenden. Die Nutzung der AGB liegt in der Regel allein im Interesse des Verwenders.24.11.2016"
                ]
            }
        },
        "11": {
            "question": "Ist ein Impressum auf einer Homepage Pflicht?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Die Impressumspflicht gilt f\u00fcr elektronische Medien, die wenigstens viermal im Kalenderjahr in vergleichbarer Gestaltung verbreitet werden, z.B. elektronische Newsletter. Umgangssprachlich werden die Offenlegungspflichten nach dem Mediengesetz h\u00e4ufig als Impressum bezeichnet!01.01.2022"
                ]
            }
        },
        "12": {
            "question": "Wann ist ein Impressum n\u00f6tig?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Jeder, der Telemedien gesch\u00e4ftsm\u00e4\u00dfig oder journalistisch-redaktionell nutzt, ben\u00f6tigt ein Impressum. Dazu z\u00e4hlen jedoch nicht nur Unternehmen, Beh\u00f6rden, Stiftungen, Vereine oder Selbstst\u00e4ndige. Auch bei einem \u00f6ffentlichen Blog oder einer Webseite, die sich durch Werbeeinnahmen finanziert, braucht man ein Impressum.16.10.2020"
                ]
            }
        },
        "13": {
            "question": "Was ist Pflicht bei einer Website?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "In Deutschland ist die Pflicht zu einem Impressum bzw. einer Anbieterkennzeichnung auf \u00a7 5 des Telemediengesetzes (TMG) und \u00a7 18 des Medienstaatsvertrags (MStV) zur\u00fcckzuf\u00fchren. Diese Paragrafen sollen sicherstellen, dass Nutzern zumindest grundlegende Informationen zum Website-Betreiber verf\u00fcgbar sind."
                ]
            }
        },
        "14": {
            "question": "Was muss rechtlich auf eine Website?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Welche Angaben das Impressum beinhalten muss, hat der Gesetzgeber \u00fcberwiegend in \u00a7 5 TMG festgelegt. So muss die Identit\u00e4t des Website-Betreibers preisgegeben werden, also bei einer juristischen Person etwa die Firma, die Rechtsform, die Anschrift und der vollst\u00e4ndige Name des/der Vertretungsberechtigten.03.12.2014"
                ]
            }
        }
    },
    "bitcoin": {
        "0": {
            "question": "How long does it take to mine 1 Bitcoin?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "The average time for generating one Bitcoin is about 10 minutes, but this applies only to powerful machines. The speed of mining depends on the type of Bitcoin mining hardware you are using."
                ]
            }
        },
        "1": {
            "question": "How does Bitcoin make money?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Key Takeaways. By mining, you can earn cryptocurrency without having to put down money for it. Bitcoin miners receive bitcoin as a reward for completing \"blocks\" of verified transactions, which are added to the blockchain."
                ]
            }
        },
        "2": {
            "question": "How do beginners buy bitcoins?",
            "answer": {
                "heading": [
                    "Follow these five steps to buy Bitcoin and keep it secure:"
                ],
                "list": [
                    "Choose a Bitcoin Exchange. Buying Bitcoin requires exchanging your dollars for some of those sweet, sweet golden coins. ...",
                    "Create Your Exchange Account. ...",
                    "Fund Your Exchange Account. ...",
                    "Order Your Bitcoin. ...",
                    "Store Your Bitcoin."
                ],
                "anwerType": "list"
            }
        },
        "3": {
            "question": "Who is Bitcoin owned by?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "who controls Bitcoin? Bitcoin is controlled by all Bitcoin users around the world. Developers are improving the software but they can't force a change in the rules of the Bitcoin protocol because all users are free to choose what software they use."
                ]
            }
        },
        "4": {
            "question": "What is the minimum amount to invest in Bitcoin?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Search for Bitcoin \u2013 Type 'BTC' or 'Bitcoin' into the search bar at the top of your screen and click 'Trade'. Invest \u2013 Enter the amount you'd like to invest in Bitcoin (minimum of $10), double-check everything is correct, and click 'Open Trade'.12.03.2022"
                ]
            }
        },
        "5": {
            "question": "How many bitcoins are left?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "How Many Bitcoins are Left to Mine? How many of the 21 million Bitcoins are left? There are 2.3 million Bitcoin left to be mined. Surprisingly, even though 18.6 million Bitcoin were mined in just over 10 years, it will take another 120 years to mine the remaining 2.3 million."
                ]
            }
        },
        "6": {
            "question": "How can I get 1 Bitcoin for free?",
            "answer": {
                "heading": [
                    "Methods To Earn Free Bitcoins"
                ],
                "list": [
                    "#1) Pionex \u2013 Using Crypto Trading Bots.",
                    "#2) Bitstamp \u2013 Using Staking Rewards.",
                    "#3) Tipping Bots And Platforms.",
                    "#4) Playing Online and Offline Games.",
                    "#5) Mining Browsers And Free Mining Software.",
                    "#6) Earning Free Bitcoins Through Bounties.",
                    "#7) Earn From Crypto Airdrops."
                ],
                "anwerType": "list"
            }
        },
        "7": {
            "question": "How much do bitcoin miners make a day?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Bitcoin mining is still extremely profitable in 2022. Bitcoin miners are currently mining around $20 million worth of Bitcoin per day. That's $600 million per month. A mining machine costs $2,000-$20,000, making it difficult for anyone but professional miners to mine.03.08.2022"
                ]
            }
        },
        "8": {
            "question": "Is it worth buying 100 dollars of Bitcoin?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Ultimately, it's up to you whether investing $100 in Bitcoin is worth it or not. If it's a one-time investment and you just want to try crypto out, we would recommend going with a lower amount since you can't profit much from $100 anyway.30.05.2022"
                ]
            }
        },
        "9": {
            "question": "Can Bitcoin be converted to cash?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "You can use a crypto exchange like Coinbase, Binance, Gemini or Kraken to turn Bitcoin into cash. This may be an easy method if you already use a centralized exchange and your crypto lives in a custodial wallet. Choose the coin and amount you'd like to sell, agree to the rates and your cash will be available to you.30.05.2022"
                ]
            }
        },
        "10": {
            "question": "Can I invest $5 in Bitcoin?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "New users get a $5 credit to invest in Bitcoin via the Bitcoin Investment Trust (GBTC), or any of the 1000+ stocks and ETFs we offer."
                ]
            }
        },
        "11": {
            "question": "How can I sell bitcoins for cash?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "To sell your bitcoin, you just need to go to an exchange and click to sell your bitcoin. Then you can choose how much you want to sell and move your cash to a bank. If you want to turn it straight to cash instead, then you should go through a bitcoin ATM.11.07.2022"
                ]
            }
        },
        "12": {
            "question": "Who gets the money when you buy Bitcoin?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "A buyer and seller agree on a price and a trade is executed over an exchange. So our $50k investor buys that amount of bitcoins and the seller receives the $50k in the form of a cash deposit. That seller may now keep it in the bank, buy other cryptos or withdraw it and spend it in any way they choose.27.11.2017"
                ]
            }
        },
        "13": {
            "question": "Is Bitcoin a good investment 2022?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Bitcoin's price has continued to fall through the first half of 2022, as has the stock market. Investors have turned to assets that perform well in a time of a slowing economy, higher inflation and rising interest rates.22.07.2022"
                ]
            }
        },
        "14": {
            "question": "Is Bitcoin still a good investment 2022?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Experts remain conflicted on whether bitcoin has bottomed out yet. Some say it already has, while others says bitcoin could fall as low as $10,000 in 2022. This volatility is a big part of why experts recommend keeping your crypto investments to less than 5% of your portfolio to begin with.22.07.2022"
                ]
            }
        }
    },
    "chicken": {
        "0": {
            "question": "Is chicken a dinosaur?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "So, are chickens dinosaurs? No \u2013 the birds are a distinct group of animals, but they did descend from the dinosaurs, and it's not too much of a twist of facts to call them modern dinosaurs. There are many similarities between the two types of animal, largely to do with bone structure."
                ]
            }
        },
        "1": {
            "question": "Is chicken a real animal?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "That seems at least partly true, but new research from Uppsala University now shows that the wild origins of the chicken are more complicated. The researchers mapped the genes that give most domesticated chickens yellow legs and found the genetic heredity derives from a closely related species, the gray jungle fowl.29.02.2008"
                ]
            }
        },
        "2": {
            "question": "What are the 3 types of chicken?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Chickens. Many different breeds of chickens have been developed for different purposes. For simplicity, you can place them into three general categories: Laying, meat-producing and dual-purpose breeds."
                ]
            }
        },
        "3": {
            "question": "What is small chicken called?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Miniature, or \u201cBantam,\u201d chickens are a subspecies of chicken that are much smaller than regular chickens. The average chicken weighs 5 to 10 pounds, whereas bantams only weigh 1 to 2.5 pounds! Most chicken breeds have a Bantam or miniature counterpart."
                ]
            }
        },
        "4": {
            "question": "Why are male chickens killed?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Male chicks are killed for two reasons: they cannot lay eggs and they are not suitable for chicken-meat production. This is because layer hens \u2014 and therefore their chicks \u2014 are a different breed of poultry to chickens that are bred and raised for meat production.22.09.2021"
                ]
            }
        },
        "5": {
            "question": "How big is a chicken brain?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "A chicken's brain is relatively small given the size of the head. It is the size of about 2 shelled peanuts, or a little bigger than a chicken's eyeball."
                ]
            }
        },
        "6": {
            "question": "Can chickens change gender?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "It is possible for a female chicken to take on external characteristics of a male, a phenomenon referred to as sex reversal."
                ]
            }
        },
        "7": {
            "question": "Is a hen a girl or boy?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Just as we humans are a particular species within mammals. And just as humans are divided into male and female and children and adults, chickens are likewise. In this context, a man equals a \u201crooster\u201d and a woman equals a \u201chen\u201d. Both are generally chickens, but the names indicate their gender and that they are adults.26.04.2019"
                ]
            }
        },
        "8": {
            "question": "Are there male chickens?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "The answer is yes, all male chickens are roosters. Roosters present a challenge to many backyard chicken owners because local jurisdictions usually have ordinances against roosters because they love to crow, which disturbs the neighbors.02.08.2022"
                ]
            }
        },
        "9": {
            "question": "Why are chickens called chooks?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "A domestic fowl; a chicken. Chook comes from British dialect chuck(y) 'a chicken; a fowl' which is a variant of chick. Chook is the common term for the live bird, although chook raffles, held in Australian clubs and pubs, have ready-to-cook chooks as prizes."
                ]
            }
        },
        "10": {
            "question": "What is a neutered chicken called?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "A capon is a male chicken that is gelded, or castrated, at a young age, and then fed a rich diet of milk or porridge. Larger than a chicken, a bit smaller than a turkey, but more flavorful than either, capons are full breasted with tender, juicy, flavorful meat that is well suited to roasting."
                ]
            }
        },
        "11": {
            "question": "Do dinosaurs Still Exist?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Other than birds, however, there is no scientific evidence that any dinosaurs, such as Tyrannosaurus, Velociraptor, Apatosaurus, Stegosaurus, or Triceratops, are still alive. These, and all other non-avian dinosaurs became extinct at least 65 million years ago at the end of the Cretaceous Period."
                ]
            }
        },
        "12": {
            "question": "What came first chicken or egg?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Back to our original question: with amniotic eggs showing up roughly 340 million or so years ago, and the first chickens evolving at around 58 thousand years ago at the earliest, it's a safe bet to say the egg came first. Eggs were around way before chickens even existed.29.03.2018"
                ]
            }
        },
        "13": {
            "question": "What was the first bird to fly?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Archaeopteryx: The Transitional Fossil. Paleontologists view Archaeopteryx as a transitional fossil between dinosaurs and modern birds. With its blend of avian and reptilian features, it was long viewed as the earliest known bird.14.03.2018"
                ]
            }
        },
        "14": {
            "question": "Can a chicken fly?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Chickens may have wings and fluffy feathers, but they're fairly dismal fliers, often going airborne for only a few yards before landing. The reason for their poor flight isn't as rhetorical as why they crossed the road.08.12.2016"
                ]
            }
        }
    }
}

Sorry, didn't see that.

Thank you so much for helping out.

 

[Thunder Digitals]

Here is base for uploading posts to wordpress using python + xmlrpc. This contains a list of 2 sample questions and answers. Questions can also be looped from csv, excel or directly after scraping.

I downloaded picture to the same folder where this script is and named it picture.jpg. Script will look image named picture.jpg from root. I have another script that selects random image from folder. Its also possible to scrape some image site using keyword. If you have more than 2 questions it will use the same image on all of those. So this will need to be modified how you want it to work.

import mimetypes
import os

from wordpress_xmlrpc import Client, WordPressPost
from wordpress_xmlrpc.methods.posts import GetPosts, NewPost
from wordpress_xmlrpc.methods.users import GetUserInfo
from wordpress_xmlrpc.compat import xmlrpc_client
from wordpress_xmlrpc.methods import media, posts

id = 'admin' # WP username
password = '123456' # WP user password
wpClient = Client('https://domain.com/xmlrpc.php', id, password) # Change domain name but leave the xmlrpc.php in the end

PAA = [
    ['This is question 1', 'This is answer for q1. It is a long established fact that a reader will be distracted by the readable content of a page when looking at its layout. The point of using Lorem Ipsum is that it has a more-or-less normal distribution of letters, as opposed to using "Content here, content here", making it look like readable English. Many desktop publishing packages and web page editors now use Lorem Ipsum as their default model text, and a search for "lorem ipsum" will uncover many web sites still in their infancy. Various versions have evolved over the years, sometimes by accident, sometimes on purpose injected humour and the like.'],
    ['This is question 2', 'This is answer for q2. Contrary to popular belief, Lorem Ipsum is not simply random text. It has roots in a piece of classical Latin literature from 45 BC, making it over 2000 years old. Richard McClintock, a Latin professor at Hampden-Sydney College in Virginia, looked up one of the more obscure Latin words, consectetur, from a Lorem Ipsum passage, and going through the cites of the word in classical literature, discovered the undoubtable source. Lorem Ipsum comes from sections 1.10.32 and 1.10.33 of "de Finibus Bonorum et Malorum" (The Extremes of Good and Evil) by Cicero, written in 45 BC. This book is a treatise on the theory of ethics, very popular during the Renaissance. The first line of Lorem Ipsum, "Lorem ipsum dolor sit amet..", comes from a line in section 1.10.32.']
       ]

def addImage(image, altText):
    imgPath = os.getcwd() + '/' + image
    print("Starting to add image to post")
    pictureMime = mimetypes.guess_type(imgPath)[0]
   
    # prepare metadata
    data = {
        'name': image,
        'type': pictureMime,  # mimetype
    }

    # read the binary file and let the XMLRPC library encode it into base64
    with open(imgPath, 'rb') as img:
        data['bits'] = xmlrpc_client.Binary(img.read())

    response = wpClient.call(media.UploadFile(data))
    # response == {
    #       'id': 6,
    #       'file': 'picture.jpg'
    #       'url': 'http://www.example.com/wp-content/uploads/2012/04/16/picture.jpg',
    #       'type': 'image/jpeg',
    # }
    attachment_id = response['id']
    attachment_url = response['url']
   
    return f'''
    <figure class="wp-block-image alignwide size-large">
    <img src="{attachment_url}" alt="{altText}" class="wp-image-{attachment_id}"/>
    </figure>
    '''

paragraph = ""

for x in range(len(PAA)):
    if x == 0:
        paragraph = paragraph + "<h1>" + PAA[x][0] + "</h1>"
        paragraph = paragraph + PAA[x][1]
    else:
        paragraph = paragraph + "<h2>" + PAA[x][0] + "</h2>"
        paragraph = paragraph + addImage('picture.jpg', PAA[x][0])
        paragraph = paragraph + PAA[x][1]


print(paragraph)

post_status = 'draft' # draft, publish

post = WordPressPost()
post.post_status = post_status
post.title = PAA[0][0].capitalize()
post.content = paragraph

post.terms_names = {
# 'post_tag': ['tag1', 'tag2'],
# 'category': ['category1', 'category2']
}

wpClient.call(NewPost(post))
print("Post done")

Here are the results that this code did

View attachment 219222

This is working but posting only the picture on the domain.
Not posting the post.

How do I solve it?

 

[babfo]

good job.

 

[Bugfisch]

This is working but posting only the picture on the domain.
Not posting the post.

How do I solve it?

I'm not into wordpress, but did you check your drafts?
By default, this line sets the post to draft, you need to change it to publish if you want it to go live when you create it.

post_status = 'draft' # draft, publish

 

[Floccer]

I'm not into wordpress, but did you check your drafts?
By default, this line sets the post to draft, you need to change it to publish if you want it to go live when you create it.

post_status = 'draft' # draft, publish

Thats true it sets posta to draft. I might have forget to tell about that. I previously understood the post would only get image but if that line is changed to publish it will publish the post

 

[b00bz]

Made some upgrades: Multi-threading and proxies support. With this version you can specify how many threads to run at once.

Put these files all in the same folder:

keywords.txt (contains a list of your keywords for google, one per line)

proxies.txt (insert your proxies according to the example below)

http://username:password@ip:port
socks5://username:password@ip:port

Put the below Python code in a file called paa.py in the same folder as the two other files. Make sure you've installed Python on your system. Run it like:



It will output a new file called paa.txt which is the result.

#People Also Ask Multithreaded GoogleYoinker with Proxy Support

import requests
import threading
from lxml.html import fromstring
import lxml.html
from time import time
from queue import Queue
import logging
import random

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)

max_threads = 10 # don't go too crazy
proxy_scheme = 'http' # https, ftp

keywords_list = open('keywords.txt', 'r')
output = open('paa.txt', 'w')

# List of proxies goes in proxies.txt, format one proxy per line:
# http://username:password@ip:port
# socks5://username:password@ip:port
proxies_list = open('proxies.txt', 'r')

keywords = keywords_list.readlines()
proxies = proxies_list.readlines()

if not proxies:
    proxies = None

header = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36',
        'Accept-Language': 'tr-tr,en;q=0.9',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
        }

class GoogleYoinker(threading.Thread):
   def __init__(self, queue):
      threading.Thread.__init__(self)
      self.queue = queue

   def run(self):
        print ("Starting " + self.name)
        while True:
            # Get the work from the queue and expand the tuple
            keyword = self.queue.get()
            try:
                query = keyword.strip()
                if proxies is not None:
                    proxy = {proxy_scheme: proxies[random.randint(0, len(proxies) - 1)].strip()}
                    logger.info("Using Proxy {}".format(proxy))
                    response = requests.get(f'https://www.google.com/search?q={query}&start=0', headers=header, proxies=proxy).text
                else:
                    response = requests.get(f'https://www.google.com/search?q={query}&start=0', headers=header).text
                tree = lxml.html.fromstring(response)
                nodes = tree.xpath('//@data-q')

                logger.info("Keyword Processed: {}".format(query))
                output.write(query)
                output.write(": ")
                for node in nodes:
                    output.write(node)
                    output.write(" , ")
                output.write("\n")

                logger.info('Exiting: {}'.format(self.name))
            finally:
                self.queue.task_done()


def main():
    ts = time()
    queue = Queue()

    # Create up to max_threads worker threads
    for x in range(max_threads):
        worker = GoogleYoinker(queue)

        # Setting daemon to True will let the main thread exit even though the workers are blocking
        worker.daemon = True
        worker.start()

    # Put the keywords into the queue
    for keyword in keywords:
        logger.info('Queueing keyword: {}'.format(keyword))
        queue.put((keyword.strip()))

    # Causes the main thread to wait for the queue to finish processing all the tasks
    queue.join()
    logging.info('Took %s', time() - ts)

    keywords_list.close()
    proxies_list.close()
    output.close()


if __name__ == '__main__':
    main()

@shog any chance You could modify it so it also scrapes answers? and maybe a setting to specify how many questions/answers per keyword? would really appreciate it

 

[Bugfisch]

@shog any chance You could modify it so it also scrapes answers? and maybe a setting to specify how many questions/answers per keyword? would really appreciate it

This won't work with simple requests or even js rendering requests. The answers and additional questions are loading dynamicly, when you click the questions. So there are only 2 ways - crawl with something that's imitating clicks like selenium or scrape the hidden API.
Going with Selenium is kind the easier way, cause you might have to handle cookie stuff, analysing your requests and stuff like this to get responses of the hidden API but that would be the more elegant (and way more performant) way.