[CODE] People Also Ask Scraping with Python [Upgraded code from iam_ironman]

prvncqr said:
It requires a lot of time to create a script that works as expected. Every coder knows that it's not possible to scrape thousands of PAA with just a few lines of code shared here.
Click to expand...
Well already we discus about this and selenium is the answer. With this code if you have 50 kw you can scrape more than 100 paa & for white that guys this pretty enough.

prove it or suck it.
 
Advertise on BHW
Ramazan said:
Well already we discus about this and selenium is the answer. With this code if you have 50 kw you can scrape more than 100 paa & for white that guys this pretty enough.
Click to expand...
Maybe. But, copy-pasting won't work for most people.
 
Ramazan said:
can u stop crying on my thread? Improve code or share your codes otherwise gtfo
Click to expand...
Very funny. Don't direct people in the wrong direction. People will end up at the wrong destination after losing a lot of money. Don't get cursed by such innocent people
 
Ramazan said:
Selling useless backlinks and talking about wrong direction :D Keep sell comment backlinks to people
Click to expand...
You proved you are naive. I don't want to waste my time. Keep going.
 
Ramazan said:
I saw the code on @iam_ironman post and upgraded with for loop, keyword list and writing resuls on txt file line by line. So also thanks for idea @RealDaddy

Python: 
import requests
from lxml.html import fromstring
import lxml.html

file1 = open('YourPath/PeopleAlsoAsk/keywords.txt', 'r')
file2 = open('YourPath/PeopleAlsoAsk/paa.txt', 'w')
Lines = file1.readlines()
header = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36',
        'Accept-Language': 'tr-tr,en;q=0.9',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
        }
for line in Lines:
        query = line.strip()
        response = requests.get(f'https://www.google.com/search?q={query}&start=0', headers=header).text
        tree = lxml.html.fromstring(response)
        node = tree.xpath('//@data-q')
        x = node[0]
        y = node[1]
        z = node[2]
        print(query)
        file2.write(query)
        file2.write(": ")
        file2.write(x)
        file2.write(" , ")
        file2.write(y)
        file2.write(" , ")
        file2.write(z)
        file2.write("\n")
file1.close()
file2.close()

Create keywords.txt file and paa.txt program is saving results so paa.txt so write your keywords line by line on keywords.txt
Example paa.txt
View attachment 218623
Click to expand...
Is there a way to scrap the answer with the questions?
 
I got inspired and created a more advanced version, wich can mostly scrape all questions and it's answers. You will need seleniumwire to run it, though.
Disclaimer: It's still very buggy and dirty coded - but serves as a start. It can also only handle text- and list answers, yet, no tables for example.
Can be used with proxies (thats why seleniumwire, not just selenium) or VPN even , most effective with some IP changing stuff like surfshark and of course be upgraded with captcha breakers or something like this, multithreaded with joblib, etc. You're free to edit it in any way ;)
Python: 
from seleniumwire import webdriver
from parsel import Selector
import random
import time

##################################################################
##################################################################
##################################################################
# Your Keyword List
kws = ["webscraping", "bitcoin", "chicken"]

# Limit scraped Questions ||| 0 == All Possible
limit = 15

##################################################################
##################################################################
##################################################################
##### FUNCTIONS
def clickyThingyXpath(xpath, wait, timeout, desc):
    startTime = time.time()
    time.sleep(wait)
    while True:
        checkTime = time.time() - timeout

        if startTime < checkTime:
            print("TIMEOUT: clickyThingyXpath["+xpath+"]")
            raise Exception("TIMEOUT: clickyThingyXpath["+xpath+"]")
        try:
            driver.find_element_by_xpath(xpath).click()
            return("SUCCESS: "+desc)
        except:
            time.sleep(1)
            pass
def findTexts(xpath, sleep):
    time.sleep(sleep)
    texts = list()
    _texts = driver.find_elements_by_xpath(xpath)
    for t in _texts:
        if len(t.text) > 5:
            texts.append(t.text)
    return(texts)

def findQuesionID(questions):
    _id = driver.find_element_by_xpath('//*[text() = "'+questions+'"]/parent::*/parent::*/parent::*/parent::*/parent::*').get_attribute("id")
    return(_id)   

def XPathByQuestionID(_id):
    xpath = '//*[@id="'+_id+'"]/div/div/div[1]/div[4]'
    return(xpath)

def XPathByAnswerID(_id):
    clicky = '//*[@id="'+_id+'"]//*[@data-attrid="wa:/description"]'
    return(clicky)

def handleAnswerList(_id):
    _dict = dict()

    _dict["heading"] = findTexts('//*[@id="'+_id+'"]//*[@role="heading"]',0)
    findList = findTexts('//*[@id="'+_id+'"]//li',0)
    if len(findList) > 0:
        _dict["list"] = findList
        _dict["anwerType"] = "list"
        return(_dict)
    else:
        return(None)

def doWhateverYouWandwithThe(results):
    import json
    with open("results.json", "w") as jsonFile:
        json.dump(results, jsonFile, indent=4)

##################################################################
##################################################################
##################################################################
##### Script

results = dict()
for kw in kws:
    results[kw] = dict()
    ### Setting Up Webdriver
    ua = ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36","Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36"]
    options = webdriver.ChromeOptions()
    chrome_prefs = {}
    options.experimental_options["prefs"] = chrome_prefs
    chrome_prefs["profile.default_content_settings"] = {"images": 2}
    chrome_prefs["profile.managed_default_content_settings"] = {"images": 2}
    options.add_argument("--disable-web-security")
    options.add_argument("--disable-site-isolation-trials")
    options.add_argument("--disable-application-cache")
    options.add_experimental_option('excludeSwitches', ['enable-logging'])
    options.add_argument('--user-agent='+random.choice(ua))
    driver = webdriver.Chrome(executable_path=r"chromedriver.exe", chrome_options=options)

    ### Request
    url = 'https://www.google.com/search?q='+kw+'&start=0'
    driver.get(url)

    ### Click on Privacy Stuff - Optional
    # clickyThingyXpath('/html/body/div[3]/div[3]/span/div/div/div/div[3]/div[1]/button[1]/div', 1, 60, "Handling Privacy Popup")

    ### Scraping
    seen = list()
    questions = findTexts("//*[@data-q]//div[@role='button']//div/span", 0)
    i = 0
    while True:
        for q in questions:
            try:
                # Preparing result dict
                results[kw][str(i)] = {"question":{}, "answer":{}}
                # check if question done allready, since in loop
                if q in seen:
                    continue
                # Get the question ID for further xpath usage
                qid = findQuesionID(q)
                print(kw+" ||| "+q)
                 # Open question
                clickyThingyXpath(XPathByQuestionID(qid), 0, 10, 'Clicking Question: "'+q+'"')
                # Scrape answer
                answer = findTexts(XPathByAnswerID(qid), 2)
                # check if answer is text only
                if len(answer) == 1:
                    anwerType = "Text"
                    while answer[0] == "":
                        answer = findTexts(XPathByAnswerID(qid), 2)
                    # Add to result dict
                    results[kw][str(i)]["question"] = q
                    results[kw][str(i)]["answer"]["answerType"] = "text"
                    results[kw][str(i)]["answer"]["answer"] = answer
                else:
                    # Not Text, so handle as list
                    answer = handleAnswerList(qid)
                    # Ignore if it's not a List as well
                    if answer == None:
                        seen.append(q)
                        continue
                    # Add to result dict
                    results[kw][str(i)]["question"] = q
                    results[kw][str(i)]["answer"] = answer
                seen.append(q)
                i+=1
                if i == limit:
                    break
            except:
                seen.append(q)
                continue
        if i == limit:
            break
        ### Did more questions pop up?
        _questions = findTexts("//*[@data-q]//div[@role='button']//div/span", 0)
        if len(questions) == len(_questions):
            break
        questions = _questions

    driver.close()

doWhateverYouWandwithThe(results)

Results Example:
JSON: 
{
    "webscraping": {
        "0": {
            "question": "What is Webscraping?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Web scraping is the process of using bots to extract content and data from a website. Unlike screen scraping, which only copies pixels displayed onscreen, web scraping extracts underlying HTML code and, with it, data stored in a database. The scraper can then replicate entire website content elsewhere."
                ]
            }
        },
        "1": {
            "question": "Is it legal to scrape Wikipedia?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Also, APIs can be surprisingly hard to use. Fortunately, we are still allowed to scrape the single greatest repository of human knowledge in history: Wikipedia.17.12.2017"
                ]
            }
        },
        "2": {
            "question": "What is web scraping in Python?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Web scraping is the process of collecting and parsing raw data from the Web, and the Python community has come up with some pretty powerful web scraping tools. The Internet hosts perhaps the greatest source of information\u2014and misinformation\u2014on the planet."
                ]
            }
        },
        "3": {
            "question": "What is required for web scraping?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Web scraping is software-based. It requires you to know at least one of the popular programming languages like Python or Ruby. These days it's hard for anyone who wants to make money online not to have these two in their toolbox. If you choose to learn Python, then you will need to learn about the BeutifulSoup library.26.11.2021"
                ]
            }
        },
        "4": {
            "question": "Is scraping data legal?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "So is it legal or illegal? Web scraping and crawling aren't illegal by themselves. After all, you could scrape or crawl your own website, without a hitch. Startups love it because it's a cheap and powerful way to gather data without the need for partnerships."
                ]
            }
        },
        "5": {
            "question": "Who uses Webscraping?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "As a result, many businesses use web scraping to improve their services and operations. For example, web scraping is incredibly common in the real estate industry for market analysis and to build databases of available real estate listing. Have you ever used one of those convenient comparison shopping websites?17.02.2020"
                ]
            }
        },
        "6": {
            "question": "Is Beautiful Soup legal?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "For example, it is legal when the data extracted is composed of directories and telephone listing for personal use. However, if the extracted data is for commercial use\u2014without the consent of the owner\u2014this would be illegal."
                ]
            }
        },
        "7": {
            "question": "Is scraping YouTube legal?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Most data found on YouTube is accessible to the general public, making it legal to scrape. But it's still important to comply with regulations that deal with personal data and copyright protection. To learn more about the legal context of web scraping, check out our blog article on the subject.22.06.2021"
                ]
            }
        },
        "8": {
            "question": "Is web scraping profitable?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Become a data engineer\n\nA web scraper at the top of his career can earn up to $131,500 annually. If you are looking for a quick and easy way to scrape websites, try our web scrapers for free!"
                ]
            }
        },
        "9": {
            "question": "Is it legal to scrape a website?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Web scraping is legal if you scrape data publicly available on the internet. But some kinds of data are protected by international regulations, so be careful scraping personal data, intellectual property, or confidential data.28.04.2022"
                ]
            }
        },
        "10": {
            "question": "Which language is best for web scraping?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Python. Python is mostly known as the best web scraper language. It's more like an all-rounder and can handle most of the web crawling-related processes smoothly. Beautiful Soup is one of the most widely used frameworks based on Python that makes scraping using this language such an easy route to take.09.08.2017"
                ]
            }
        },
        "11": {
            "question": "Does Amazon allow web scraping?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Amazon can detect Bots and block their IPs\n\nSince Amazon prevents web scraping on its pages, it can easily detect if an action is being executed by a scraper bot or through a browser by a manual agent.05.02.2021"
                ]
            }
        },
        "12": {
            "question": "How long does it take to learn web scraping?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "It takes one week to learn the basics of web development technologies. One week to learn web scraping and python libraries like NumPy, pandas, matplotlib for data handling and analysis.11.04.2020"
                ]
            }
        },
        "13": {
            "question": "Do I need to know HTML for web scraping?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "It's not hard to understand, but before you can start web scraping, you need to first master HTML. To extract the right pieces of information, you need to right-click \u201cinspect.\u201d You'll find a very long HTML code that seems infinite. Don't worry. You don't need to know HTML deeply to be able to extract the data.12.04.2021"
                ]
            }
        },
        "14": {
            "question": "How much does web scraping cost?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "A web scraping team is made up of technical gurus that come together to create a web scraping agency. For a team service, the web scraping cost might be high or low depending on the size of the job. The cost usually ranges from around $600 to $1000.09.03.2022"
                ]
            }
        }
    },
    "bitcoin": {
        "0": {
            "question": "How does Bitcoin make money?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Key Takeaways. By mining, you can earn cryptocurrency without having to put down money for it. Bitcoin miners receive bitcoin as a reward for completing \"blocks\" of verified transactions, which are added to the blockchain."
                ]
            }
        },
        "1": {
            "question": "Are Bitcoin a good investment?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "You can easily trade bitcoin for cash or assets like gold instantly with incredibly low fees. The high liquidity associated with bitcoin makes it a great investment vessel if you're looking for short-term profit. Digital currencies may also be a long-term investment due to their high market demand.02.07.2022"
                ]
            }
        },
        "2": {
            "question": "How do Bitcoins work?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Bitcoin is a form of digital cash that eliminates the need for central authorities such as banks or governments. Instead, Bitcoin uses a peer-to-peer internet network to confirm purchases directly between users."
                ]
            }
        },
        "3": {
            "question": "How long does it take to mine 1 Bitcoin?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "The average time for generating one Bitcoin is about 10 minutes, but this applies only to powerful machines. The speed of mining depends on the type of Bitcoin mining hardware you are using."
                ]
            }
        },
        "4": {
            "question": "Is Bitcoin real money?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Bitcoin is a decentralized digital currency that you can buy, sell and exchange directly, without an intermediary like a bank. Bitcoin's creator, Satoshi Nakamoto, originally described the need for \u201can electronic payment system based on cryptographic proof instead of trust.\u201d08.06.2022"
                ]
            }
        },
        "5": {
            "question": "How do beginners invest in Bitcoins?",
            "answer": {
                "heading": [
                    "Here's how to invest in Bitcoin, in 5 easy steps: Join a Bitcoin Exchange. Get a Bitcoin Wallet.\n..."
                ],
                "list": [
                    "Join a Bitcoin Exchange. ...",
                    "Get a Bitcoin Wallet. ...",
                    "Connect Your Wallet to a Bank Account. ...",
                    "Place Your Bitcoin Order. ...",
                    "Manage Your Bitcoin Investments."
                ],
                "anwerType": "list"
            }
        },
        "6": {
            "question": "Are Bitcoins illegal?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "If the patchwork of regulation confuses you, here's the bottom line. Bitcoin is not illegal in the U.S. How you can buy it, what services and exchanges you can use and what you can use it for might depend on which state you are in, however.10.03.2022"
                ]
            }
        },
        "7": {
            "question": "What coin should I buy now?",
            "answer": {
                "heading": [
                    "7 best cryptocurrencies to buy now:"
                ],
                "list": [
                    "Bitcoin (BTC)",
                    "Ether (ETH)",
                    "Solana (SOL)",
                    "Avalanche (AVAX)",
                    "Binance Coin (BNB)",
                    "Tron (TRX)",
                    "Cosmos (ATOM)"
                ],
                "anwerType": "list"
            }
        },
        "8": {
            "question": "Is Bitcoin worth investing in 2021?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Bitcoin's Future Outlook\n\nBitcoin is a good indicator of the crypto market in general, because it's the largest cryptocurrency by market cap and the rest of the market tends to follow its trends. Bitcoin's price had a wild ride in 2021, and last November set another new all-time high price when it went over $68,000.vor 5 Tagen"
                ]
            }
        },
        "9": {
            "question": "Can you get rich off of Bitcoin?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "There's no denying that some cryptocurrency traders have become millionaires thanks to their successful investments. What's not as often discussed is the great number of people who have lost significant sums trying to become rich by investing in crypto.13.06.2022"
                ]
            }
        },
        "10": {
            "question": "Who gets the money when you buy Bitcoin?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "A buyer and seller agree on a price and a trade is executed over an exchange. So our $50k investor buys that amount of bitcoins and the seller receives the $50k in the form of a cash deposit. That seller may now keep it in the bank, buy other cryptos or withdraw it and spend it in any way they choose.27.11.2017"
                ]
            }
        },
        "11": {
            "question": "Are bitcoins safe?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Although Bitcoin uses secure cryptography, you could argue it's not a safe investment because of its volatility. With no regulatory body and an international, 24/7 market, a bitcoin worth $60,000 one day can be worth $30,000 just a few days later. Though there have been some periods of stability, these never last long.02.06.2022"
                ]
            }
        },
        "12": {
            "question": "How many bitcoin are left?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "How Many Bitcoins are Left to Mine? How many of the 21 million Bitcoins are left? There are 2.3 million Bitcoin left to be mined. Surprisingly, even though 18.6 million Bitcoin were mined in just over 10 years, it will take another 120 years to mine the remaining 2.3 million."
                ]
            }
        },
        "13": {
            "question": "How can I get 1 bitcoin for free?",
            "answer": {
                "heading": [
                    "Methods To Earn Free Bitcoins"
                ],
                "list": [
                    "#1) Pionex \u2013 Using Crypto Trading Bots.",
                    "#2) Bitstamp \u2013 Using Staking Rewards.",
                    "#3) Tipping Bots And Platforms.",
                    "#4) Playing Online and Offline Games.",
                    "#5) Mining Browsers And Free Mining Software.",
                    "#6) Earning Free Bitcoins Through Bounties.",
                    "#7) Earn From Crypto Airdrops."
                ],
                "anwerType": "list"
            }
        },
        "14": {
            "question": "How much do bitcoin miners make a day?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Mining Revenue\n\nIn February 2022, one Bitcoin mining machine (commonly known as an ASIC), like the Whatsminer M20S, generates around $12 in Bitcoin revenue every day depending on the price of bitcoin.07.07.2022"
                ]
            }
        }
    },
    "chicken": {
        "0": {
            "question": "How chicken is not good for health?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Chicken products contain cholesterol, carcinogens, and contaminants. Cholesterol, carcinogens, pathogens, and even feces found in chicken products increase the risk of heart disease, breast and prostate cancers, urinary tract infections, and foodborne illness."
                ]
            }
        },
        "1": {
            "question": "What is female chicken called?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Hen - A female chicken over one year or age. Inbred - The offspring of closely related parents; resulting from inbreeding. Incrossbred - The offspring from crossing inbred parents of the same or different breeds. Layers - Mature female chickens kept for egg production; also called laying hens."
                ]
            }
        },
        "2": {
            "question": "Is chicken a real animal?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "That seems at least partly true, but new research from Uppsala University now shows that the wild origins of the chicken are more complicated. The researchers mapped the genes that give most domesticated chickens yellow legs and found the genetic heredity derives from a closely related species, the gray jungle fowl.29.02.2008"
                ]
            }
        },
        "3": {
            "question": "What are the side effects of chicken?",
            "answer": {
                "heading": [
                    "4 side effects of eating chicken daily that you need to know..."
                ],
                "list": [
                    "01/5Can chicken cause any side effects? Chicken is probably the most popular pick among non-veg lovers. ...",
                    "02/5Can increase cholesterol. ...",
                    "03/5High heat food. ...",
                    "04/5Weight gain. ...",
                    "05/5Linked to UTIs."
                ],
                "anwerType": "list"
            }
        },
        "4": {
            "question": "Which meat is healthiest?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Liver. Liver, particularly beef liver, is one of the most nutritious meats you can eat. It's a great source of high-quality protein; vitamins A, B12, B6; folic acid; iron; zinc; and essential amino acids.17.08.2020"
                ]
            }
        },
        "5": {
            "question": "How do you tell if a chicken is a boy or girl?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "The males have pointed feathers around the neck, back, and tail; in females these feathers have round ends. If the chickens are purebreds, the coloring patterns of the males and females will also differ. In addition, males typically have larger combs and wattles and large spurs on the back of the shank (leg)."
                ]
            }
        },
        "6": {
            "question": "Do male chickens lay eggs?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Because male chickens do not lay eggs and only those in breeding programmes are required to fertilise eggs, they are considered redundant to the egg-laying industry and are usually killed shortly after being sexed, which occurs just days after they are conceived or after they hatch."
                ]
            }
        },
        "7": {
            "question": "Do chickens fly?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Chickens may have wings and fluffy feathers, but they're fairly dismal fliers, often going airborne for only a few yards before landing. The reason for their poor flight isn't as rhetorical as why they crossed the road.08.12.2016"
                ]
            }
        },
        "8": {
            "question": "Why do we not eat male chickens?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Male chicks are killed for two reasons: they cannot lay eggs and they are not suitable for chicken-meat production. This is because layer hens \u2014 and therefore their chicks \u2014 are a different breed of poultry to chickens that are bred and raised for meat production.22.09.2021"
                ]
            }
        },
        "9": {
            "question": "Do chickens lay eggs without a rooster?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Hens will lay eggs with or without a rooster. Without a rooster, your hens' eggs are infertile, so won't develop into chicks. If you do have a rooster, eggs need to be collected daily and kept in a cool place before being used so that they won't develop into chicks.15.06.2021"
                ]
            }
        },
        "10": {
            "question": "Is a chicken a bird yes or no?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "A chicken is a bird. One of the features that differentiate it from most other birds is that it has a comb and two wattles. The comb is the red appendage on the top of the head, and the wattles are the two appendages under the chin."
                ]
            }
        },
        "11": {
            "question": "Are chickens a dinosaur?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "So, are chickens dinosaurs? No \u2013 the birds are a distinct group of animals, but they did descend from the dinosaurs, and it's not too much of a twist of facts to call them modern dinosaurs. There are many similarities between the two types of animal, largely to do with bone structure."
                ]
            }
        },
        "12": {
            "question": "Are chicken man made?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Prolific egg-laying chickens have been created by humans through hundreds of years of selective breeding.21.11.2013"
                ]
            }
        },
        "13": {
            "question": "What are disadvantages of eating chicken?",
            "answer": {
                "heading": [
                    "Side Effects of Eating Chicken, Says Science"
                ],
                "list": [
                    "It may increase your cholesterol levels.",
                    "The majority of retail chicken breast is contaminated with bacteria.",
                    "It can lead to weight gain.",
                    "Chicken with antibiotics has been linked to UTIs."
                ],
                "anwerType": "list"
            }
        },
        "14": {
            "question": "Is eating chicken everyday healthy?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "In addition to helping you build muscle mass, eating chicken every day can preserve and protect your overall muscle health, as noted by Mazzuco. \"Chicken breast is also a good source of leucine, which is vital for muscle growth, muscle repair, and improving endurance and strength,\" she continued.13.04.2021"
                ]
            }
        }
    }
}
 
Bugfisch said:
I got inspired and created a more advanced version, wich can mostly scrape all questions and it's answers. You will need seleniumwire to run it, though.
Disclaimer: It's still very buggy and dirty coded - but serves as a start. It can also only handle text- and list answers, yet, no tables for example.
Can be used with proxies (thats why seleniumwire, not just selenium) or VPN even , most effective with some IP changing stuff like surfshark and of course be upgraded with captcha breakers or something like this, multithreaded with joblib, etc. You're free to edit it in any way ;)
Click to expand...
Thanks for support this thread :)
 
Bugfisch said:
I got inspired and created a more advanced version, wich can mostly scrape all questions and it's answers. You will need seleniumwire to run it, though.
Disclaimer: It's still very buggy and dirty coded - but serves as a start. It can also only handle text- and list answers, yet, no tables for example.
Can be used with proxies (thats why seleniumwire, not just selenium) or VPN even , most effective with some IP changing stuff like surfshark and of course be upgraded with captcha breakers or something like this, multithreaded with joblib, etc. You're free to edit it in any way ;)
Click to expand...
it does not work for me . its gives error like this from antivurus i guess. ı closed antivirus but still nothing changed.

https://prnt.sc/gOowuygkuna1
 
prvncqr said:
It requires a lot of time to create a script that works as intended. Every experienced coder knows that it's not possible to scrape thousands of PAA with just a few lines of code shared here. OP is indirectly trying to promote his service.
You must have gotten at least 10 PMs by now, I suppose. Best of luck. :)
Click to expand...
I somewhat agree;
With the code in the OP probably not as you'd eventually be rate limited, however with some tweaks as others have done it has a lot of potential.
Still nice work OP.
 
Amazing thread!
I am just curious why I didn't follow it from the very beginning.
 
desperado596 said:
it does not work for me . its gives error like this from antivurus i guess. ı closed antivirus but still nothing changed.
Click to expand...
Ok, I haven't seen something like this before =D
As far as I can get of this message, my Turkish is basicly not existent, even though I live in the only left turkish community in Non-Occupied Cyprus^^
You can try commenting out this lines:
Python: 
    options.add_argument("--disable-web-security")
    options.add_argument("--disable-site-isolation-trials")
    options.add_argument("--disable-application-cache")
    options.add_experimental_option('excludeSwitches', ['enable-logging'])

I think, they're basicly not needed and might trigger securtity programms. These are mostly relics of an account faker I use, I think microsoft needed this^^
 
Bugfisch said:
Ok, I haven't seen something like this before =D
As far as I can get of this message, my Turkish is basicly not existent, even though I live in the only left turkish community in Non-Occupied Cyprus^^
You can try commenting out this lines:
Python: 
    options.add_argument("--disable-web-security")
    options.add_argument("--disable-site-isolation-trials")
    options.add_argument("--disable-application-cache")
    options.add_experimental_option('excludeSwitches', ['enable-logging'])

I think, they're basicly not needed and might trigger securtity programms. These are mostly relics of an account faker I use, I think microsoft needed this^^
Click to expand...
thanks for the answer. i deleted somethings and the script worked but its searching keywords so fast and in a few seconds google showed captcha pretty fast. so it did not work for me again .
 
Made some upgrades: Multi-threading and proxies support. With this version you can specify how many threads to run at once.

Put these files all in the same folder:

keywords.txt (contains a list of your keywords for google, one per line)

proxies.txt (insert your proxies according to the example below)
Code: 
http://username:password@ip:port
socks5://username:password@ip:port

Put the below Python code in a file called paa.py in the same folder as the two other files. Make sure you've installed Python on your system. Run it like:

python paa.py
Click to expand...

It will output a new file called paa.txt which is the result.


Python: 
#People Also Ask Multithreaded GoogleYoinker with Proxy Support

import requests
import threading
from lxml.html import fromstring
import lxml.html
from time import time
from queue import Queue
import logging
import random

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)

max_threads = 10 # don't go too crazy
proxy_scheme = 'http' # https, ftp

keywords_list = open('keywords.txt', 'r')
output = open('paa.txt', 'w')

# List of proxies goes in proxies.txt, format one proxy per line:
# http://username:password@ip:port
# socks5://username:password@ip:port
proxies_list = open('proxies.txt', 'r')

keywords = keywords_list.readlines()
proxies = proxies_list.readlines()

if not proxies:
    proxies = None

header = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36',
        'Accept-Language': 'tr-tr,en;q=0.9',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
        }

class GoogleYoinker(threading.Thread):
   def __init__(self, queue):
      threading.Thread.__init__(self)
      self.queue = queue

   def run(self):
        print ("Starting " + self.name)
        while True:
            # Get the work from the queue and expand the tuple
            keyword = self.queue.get()
            try:
                query = keyword.strip()
                if proxies is not None:
                    proxy = {proxy_scheme: proxies[random.randint(0, len(proxies) - 1)].strip()}
                    logger.info("Using Proxy {}".format(proxy))
                    response = requests.get(f'https://www.google.com/search?q={query}&start=0', headers=header, proxies=proxy).text
                else:
                    response = requests.get(f'https://www.google.com/search?q={query}&start=0', headers=header).text
                tree = lxml.html.fromstring(response)
                nodes = tree.xpath('//@data-q')

                logger.info("Keyword Processed: {}".format(query))
                output.write(query)
                output.write(": ")
                for node in nodes:
                    output.write(node)
                    output.write(" , ")
                output.write("\n")

                logger.info('Exiting: {}'.format(self.name))
            finally:
                self.queue.task_done()


def main():
    ts = time()
    queue = Queue()

    # Create up to max_threads worker threads
    for x in range(max_threads):
        worker = GoogleYoinker(queue)

        # Setting daemon to True will let the main thread exit even though the workers are blocking
        worker.daemon = True
        worker.start()

    # Put the keywords into the queue
    for keyword in keywords:
        logger.info('Queueing keyword: {}'.format(keyword))
        queue.put((keyword.strip()))

    # Causes the main thread to wait for the queue to finish processing all the tasks
    queue.join()
    logging.info('Took %s', time() - ts)

    keywords_list.close()
    proxies_list.close()
    output.close()


if __name__ == '__main__':
    main()
 
Last edited:
desperado596 said:
thanks for the answer. i deleted somethings and the script worked but its searching keywords so fast and in a few seconds google showed captcha pretty fast. so it did not work for me again .
Click to expand...
That might be your IP. You can set residential proxies or try an VPN for smaller cases.
And the script still has it's problems when no questions are there ;)

As said, it's more a starting script for Coders, than unsefull in a productional setup.

For using proxy servers:
Python: 
proxy = str("yourProxyLogin:yourProxyPW@yourProxyServerIP:yourProxyServerPort")
soptions = {
     'proxy': {'http': "http://"+proxy,'https': "http://"+proxy, 'no_proxy': "http://"+proxy}
 }

### REPLACE:
driver = webdriver.Chrome(executable_path=r"chromedriver.exe", chrome_options=options)
### WITH:
driver = webdriver.Chrome(executable_path=r"chromedriver.exe", chrome_options=options, seleniumwire_options=soptions)

edit: made a correctional change in proxy code^^
 
Today I let my daughter pretend to drive my car. She's 4.

She got really enthusiastic, and was pretty happy she was "driving".
But in the end, the car didn't move.

I hope I'm not being offtopic, I really am not.
 
Alexion said:
Today I let my daughter pretend to drive my car. She's 4.

She got really enthusiastic, and was pretty happy she was "driving".
But in the end, the car didn't move.

I hope I'm not being offtopic, I really am not.
Click to expand...
Beeing enthusiastic about st is the first step of learning. And learning is the first step of mastering ;)
Of course, these scripts - mine included, I'm a really bad coder, but I can solve problems quick and dirty xD - are not usefull for any productional purpose. But they solve some nasty starting stuff, like handling xpathes etc. And if people use it, try to understand it and gain their learning it's worth it in the end ;)
It's more about prototyping - and prototypes should be thrown away, after proof of concept. The concept is the point.
 
Bugfisch said:
Beeing enthusiastic about st is the first step of learning. And learning is the first step of mastering ;)
Of course, these scripts - mine included, I'm a really bad coder, but I can solve problems quick and dirty xD - are not usefull for any productional purpose. But they solve some nasty starting stuff, like handling xpathes etc. And if people use it, try to understand it and gain their learning it's worth it in the end ;)
It's more about prototyping - and prototypes should be thrown away, after proof of concept. The concept is the point.
Click to expand...
oh, the code is actually pretty good, especially true if you are just starting out.

Was referring to other aspects, otherwise hats off to you for doing this.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top