[CODE] People Also Ask Scraping with Python [Upgraded code from iam_ironman]

Bugfisch said:
I'm not into wordpress, but did you check your drafts?
By default, this line sets the post to draft, you need to change it to publish if you want it to go live when you create it.
Python: 
post_status = 'draft' # draft, publish
Click to expand...
Floccer said:
Thats true it sets posta to draft. I might have forget to tell about that. I previously understood the post would only get image but if that line is changed to publish it will publish the post
Click to expand...
Yes, I checked in the draft. Nothing there.

Also, changed the status from draft to publish. But still nothing.

Also have tried posting on a different domain but the results are the same, only uploading picture to WordPress.
 
Advertise on BHW
mmubeen8 said:
Yes, I checked in the draft. Nothing there.

Also, changed the status from draft to publish. But still nothing.

Also have tried posting on a different domain but the results are the same, only uploading picture to WordPress.
Click to expand...
I can check later today if there is some error in it. I did test that piece of code before I posted it so I thought it would work.
 
mmubeen8 said:
Alright. Thank you so much buddy.
Click to expand...
So i tested my code. Copy pasted from this thread and changed credentials and domain and it worked for me. It made post as draft. I don't really have a clue why it won't work.

Try this. Remove these 2 lines at the end:
Python: 
wpClient.call(NewPost(post))
print("Post done")

And replace those with this:
Python: 
print(f"Post done. ID is {wpClient.call(NewPost(post))}")

it will show the post id in the console at least if it works. I don't know what it shows if it fails.

Like this
Python: 
Post done. ID is 2353
 
Floccer said:
So i tested my code. Copy pasted from this thread and changed credentials and domain and it worked for me. It made post as draft. I don't really have a clue why it won't work.

Try this. Remove these 2 lines at the end:
Python: 
wpClient.call(NewPost(post))
print("Post done")

And replace those with this:
Python: 
print(f"Post done. ID is {wpClient.call(NewPost(post))}")

it will show the post id in the console at least if it works. I don't know what it shows if it fails.

Like this
Python: 
Post done. ID is 2353
Click to expand...
I checked the code again. And yes, it is working.

Thank you so much.
 
shog said:
Made some upgrades: Multi-threading and proxies support. With this version you can specify how many threads to run at once.

Put these files all in the same folder:

keywords.txt (contains a list of your keywords for google, one per line)

proxies.txt (insert your proxies according to the example below)
Code: 
http://username:password@ip:port
socks5://username:password@ip:port

Put the below Python code in a file called paa.py in the same folder as the two other files. Make sure you've installed Python on your system. Run it like:



It will output a new file called paa.txt which is the result.


Python: 
#People Also Ask Multithreaded GoogleYoinker with Proxy Support

import requests
import threading
from lxml.html import fromstring
import lxml.html
from time import time
from queue import Queue
import logging
import random

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)

max_threads = 10 # don't go too crazy
proxy_scheme = 'http' # https, ftp

keywords_list = open('keywords.txt', 'r')
output = open('paa.txt', 'w')

# List of proxies goes in proxies.txt, format one proxy per line:
# http://username:password@ip:port
# socks5://username:password@ip:port
proxies_list = open('proxies.txt', 'r')

keywords = keywords_list.readlines()
proxies = proxies_list.readlines()

if not proxies:
    proxies = None

header = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36',
        'Accept-Language': 'tr-tr,en;q=0.9',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
        }

class GoogleYoinker(threading.Thread):
   def __init__(self, queue):
      threading.Thread.__init__(self)
      self.queue = queue

   def run(self):
        print ("Starting " + self.name)
        while True:
            # Get the work from the queue and expand the tuple
            keyword = self.queue.get()
            try:
                query = keyword.strip()
                if proxies is not None:
                    proxy = {proxy_scheme: proxies[random.randint(0, len(proxies) - 1)].strip()}
                    logger.info("Using Proxy {}".format(proxy))
                    response = requests.get(f'https://www.google.com/search?q={query}&start=0', headers=header, proxies=proxy).text
                else:
                    response = requests.get(f'https://www.google.com/search?q={query}&start=0', headers=header).text
                tree = lxml.html.fromstring(response)
                nodes = tree.xpath('//@data-q')

                logger.info("Keyword Processed: {}".format(query))
                output.write(query)
                output.write(": ")
                for node in nodes:
                    output.write(node)
                    output.write(" , ")
                output.write("\n")

                logger.info('Exiting: {}'.format(self.name))
            finally:
                self.queue.task_done()


def main():
    ts = time()
    queue = Queue()

    # Create up to max_threads worker threads
    for x in range(max_threads):
        worker = GoogleYoinker(queue)

        # Setting daemon to True will let the main thread exit even though the workers are blocking
        worker.daemon = True
        worker.start()

    # Put the keywords into the queue
    for keyword in keywords:
        logger.info('Queueing keyword: {}'.format(keyword))
        queue.put((keyword.strip()))

    # Causes the main thread to wait for the queue to finish processing all the tasks
    queue.join()
    logging.info('Took %s', time() - ts)

    keywords_list.close()
    proxies_list.close()
    output.close()


if __name__ == '__main__':
    main()
Click to expand...
How to get PAA answers?
Where to find results.json file?
 
Bugfisch said:
It's scraping the answers, it's just not printing em in console - most of em, by now only lists and text answers, no tables or videos. You find em in the results.json or edit the doWhateverYouWandwithThe(results) function and save em in a csv, a db or whatever instead.
Code: 
{
    "webscraping": {
        "0": {
            "question": "Was ist ein Scraping Tool?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Ein Web Scraping Tool kann verschiedene Arten an Informationen extrahieren und diese wiederum in verschiedenen Dateitypen ausgeben. So k\u00f6nnen neben reinem Text beispielsweise auch Bilder und Videos oder E-Mail Adressen und URL's extrahiert werden."
                ]
            }
        },
        "1": {
            "question": "Wie funktioniert ein Web Scraper?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Beim Web Scraping (engl. scraping = \u201ekratzen/absch\u00fcrfen\u201c) werden Daten von Webseiten extrahiert und gespeichert, um diese zu analysieren oder anderweitig zu verwerten. Beim Scraping werden viele verschiedene Arten von Informationen gesammelt."
                ]
            }
        },
        "2": {
            "question": "Ist Crawlen verboten?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Die Rechtslage l\u00e4sst sich so zusammenfassen, dass Screescraping und Webscraping grunds\u00e4tzlich zul\u00e4ssig ist, wenn von den eingesetzten Bots (Crawler, Scraper) keine technischen Schutzma\u00dfnahmen \u00fcberwunden werden und man auch keine eigene \"Schattendatenbank\" mit den Daten best\u00fcckt.01.10.2020"
                ]
            }
        },
        "3": {
            "question": "Ist Web Scraping legal Schweiz?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Zum urheberrechtlichen Schutz der Listen: Datenkompilation ist trotz Aufwand und Wert grunds\u00e4tzlich nicht gesch\u00fctzt, sofern weder die Gestaltung noch die Auswahlkriterien originell sind. Manuelles Eintippen von Daten ist kein technisches Reproduktionsverfahren i.S.v. Art."
                ]
            }
        },
        "4": {
            "question": "Ist ein Impressum in der Schweiz Pflicht?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "In der Schweiz besteht bislang keine generelle Impressumspflicht f\u00fcr Websites. Ein Impressum ist deshalb nur in Ausnahmef\u00e4llen rechtlich notwendig. Nun wird aber im Fr\u00fchling 2012 eine generelle Impressumspflicht f\u00fcr den \u00abelektronischen Gesch\u00e4ftsverkehr\u00bb in der Schweiz eingef\u00fchrt.01.12.2011"
                ]
            }
        },
        "5": {
            "question": "Was muss alles auf einer Website stehen Schweiz?",
            "answer": {
                "heading": [
                    "Es sind die folgenden Angaben zu machen:"
                ],
                "list": [
                    "Vorname/Name bei nat\u00fcrlichen Personen bzw. vollst\u00e4ndige Firma bei Unternehmen.",
                    "Postadresse (Wohnsitz bzw. Sitz und weitere Adressangaben, Postfachnummer gen\u00fcgt nicht)",
                    "E-Mail-Adresse.",
                    "Optional: Telefonnummer sowie Faxnummer (sofern vorhanden)"
                ],
                "anwerType": "list"
            }
        },
        "6": {
            "question": "Was muss in der Schweiz in ein Impressum?",
            "answer": {
                "heading": [
                    "Impressumspflicht in der Schweiz"
                ],
                "list": [
                    "Ihr kompletter Name oder der Name des Unternehmens.",
                    "Ihre Adresse oder der Sitz des Unternehmens (Postfach reicht nicht aus)",
                    "E-Mail-Adresse (Kontaktformular reicht nicht aus. Ausserdem darf die E-Mail-Adresse nicht als Bild \u201egetarnt\u201c werden.)"
                ],
                "anwerType": "list"
            }
        },
        "7": {
            "question": "Was passiert wenn man kein Impressum hat?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Fast jede Webseite im Internet ben\u00f6tigt eine Anbieterkennzeichnung auch Impressum genannt. Im Impressum muss eine aktuelle Adresse angegeben werden, das Impressum muss leicht erreichbar sein. Ist kein Impressum auf einer Webseite vorhanden kann das zu teuren Abmahnungen f\u00fchren.13.07.2022"
                ]
            }
        },
        "8": {
            "question": "Was darf ich auf meiner Homepage ver\u00f6ffentlichen?",
            "answer": {
                "heading": [
                    "Man muss also f\u00fcr alle urheberrechtlich gesch\u00fctzten Werke, die auf der Website erscheinen, das Recht haben, sie zu ver\u00f6ffentlichen."
                ],
                "list": [
                    "Freie Lizenzen. Es ist erlaubt, Inhalte zu verwenden, die vom Urheber ausdr\u00fccklich zur Verwendung freigegeben sind. ...",
                    "Fotos. ...",
                    "Videos. ...",
                    "Musik. ...",
                    "Grafiken. ...",
                    "Texte. ...",
                    "Spezialfall Sch\u00fclerzeitung?"
                ],
                "anwerType": "list"
            }
        },
        "9": {
            "question": "Wer ist verantwortlich f\u00fcr eine Website?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "F\u00fcr eine Website gibt es prinzipiell immer zwei verantwortliche Stelle mit gr\u00f6\u00dftenteils disjunkten Aufgabenbereichen. Dem Betreiber obliegt die Hoheit \u00fcber den eigentlichen Inhalt der Website, also der ver\u00f6ffentlichten Informationen, der Webhoster ist f\u00fcr die (technische) Verf\u00fcgbarkeit der Website verantwortlich.29.04.2016"
                ]
            }
        },
        "10": {
            "question": "Sind AGB auf Website Pflicht?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Grunds\u00e4tzlich gibt es keine Pflicht, AGB auf einer Webseite zu verwenden. Die Nutzung der AGB liegt in der Regel allein im Interesse des Verwenders.24.11.2016"
                ]
            }
        },
        "11": {
            "question": "Ist ein Impressum auf einer Homepage Pflicht?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Die Impressumspflicht gilt f\u00fcr elektronische Medien, die wenigstens viermal im Kalenderjahr in vergleichbarer Gestaltung verbreitet werden, z.B. elektronische Newsletter. Umgangssprachlich werden die Offenlegungspflichten nach dem Mediengesetz h\u00e4ufig als Impressum bezeichnet!01.01.2022"
                ]
            }
        },
        "12": {
            "question": "Wann ist ein Impressum n\u00f6tig?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Jeder, der Telemedien gesch\u00e4ftsm\u00e4\u00dfig oder journalistisch-redaktionell nutzt, ben\u00f6tigt ein Impressum. Dazu z\u00e4hlen jedoch nicht nur Unternehmen, Beh\u00f6rden, Stiftungen, Vereine oder Selbstst\u00e4ndige. Auch bei einem \u00f6ffentlichen Blog oder einer Webseite, die sich durch Werbeeinnahmen finanziert, braucht man ein Impressum.16.10.2020"
                ]
            }
        },
        "13": {
            "question": "Was ist Pflicht bei einer Website?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "In Deutschland ist die Pflicht zu einem Impressum bzw. einer Anbieterkennzeichnung auf \u00a7 5 des Telemediengesetzes (TMG) und \u00a7 18 des Medienstaatsvertrags (MStV) zur\u00fcckzuf\u00fchren. Diese Paragrafen sollen sicherstellen, dass Nutzern zumindest grundlegende Informationen zum Website-Betreiber verf\u00fcgbar sind."
                ]
            }
        },
        "14": {
            "question": "Was muss rechtlich auf eine Website?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Welche Angaben das Impressum beinhalten muss, hat der Gesetzgeber \u00fcberwiegend in \u00a7 5 TMG festgelegt. So muss die Identit\u00e4t des Website-Betreibers preisgegeben werden, also bei einer juristischen Person etwa die Firma, die Rechtsform, die Anschrift und der vollst\u00e4ndige Name des/der Vertretungsberechtigten.03.12.2014"
                ]
            }
        }
    },
    "bitcoin": {
        "0": {
            "question": "How long does it take to mine 1 Bitcoin?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "The average time for generating one Bitcoin is about 10 minutes, but this applies only to powerful machines. The speed of mining depends on the type of Bitcoin mining hardware you are using."
                ]
            }
        },
        "1": {
            "question": "How does Bitcoin make money?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Key Takeaways. By mining, you can earn cryptocurrency without having to put down money for it. Bitcoin miners receive bitcoin as a reward for completing \"blocks\" of verified transactions, which are added to the blockchain."
                ]
            }
        },
        "2": {
            "question": "How do beginners buy bitcoins?",
            "answer": {
                "heading": [
                    "Follow these five steps to buy Bitcoin and keep it secure:"
                ],
                "list": [
                    "Choose a Bitcoin Exchange. Buying Bitcoin requires exchanging your dollars for some of those sweet, sweet golden coins. ...",
                    "Create Your Exchange Account. ...",
                    "Fund Your Exchange Account. ...",
                    "Order Your Bitcoin. ...",
                    "Store Your Bitcoin."
                ],
                "anwerType": "list"
            }
        },
        "3": {
            "question": "Who is Bitcoin owned by?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "who controls Bitcoin? Bitcoin is controlled by all Bitcoin users around the world. Developers are improving the software but they can't force a change in the rules of the Bitcoin protocol because all users are free to choose what software they use."
                ]
            }
        },
        "4": {
            "question": "What is the minimum amount to invest in Bitcoin?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Search for Bitcoin \u2013 Type 'BTC' or 'Bitcoin' into the search bar at the top of your screen and click 'Trade'. Invest \u2013 Enter the amount you'd like to invest in Bitcoin (minimum of $10), double-check everything is correct, and click 'Open Trade'.12.03.2022"
                ]
            }
        },
        "5": {
            "question": "How many bitcoins are left?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "How Many Bitcoins are Left to Mine? How many of the 21 million Bitcoins are left? There are 2.3 million Bitcoin left to be mined. Surprisingly, even though 18.6 million Bitcoin were mined in just over 10 years, it will take another 120 years to mine the remaining 2.3 million."
                ]
            }
        },
        "6": {
            "question": "How can I get 1 Bitcoin for free?",
            "answer": {
                "heading": [
                    "Methods To Earn Free Bitcoins"
                ],
                "list": [
                    "#1) Pionex \u2013 Using Crypto Trading Bots.",
                    "#2) Bitstamp \u2013 Using Staking Rewards.",
                    "#3) Tipping Bots And Platforms.",
                    "#4) Playing Online and Offline Games.",
                    "#5) Mining Browsers And Free Mining Software.",
                    "#6) Earning Free Bitcoins Through Bounties.",
                    "#7) Earn From Crypto Airdrops."
                ],
                "anwerType": "list"
            }
        },
        "7": {
            "question": "How much do bitcoin miners make a day?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Bitcoin mining is still extremely profitable in 2022. Bitcoin miners are currently mining around $20 million worth of Bitcoin per day. That's $600 million per month. A mining machine costs $2,000-$20,000, making it difficult for anyone but professional miners to mine.03.08.2022"
                ]
            }
        },
        "8": {
            "question": "Is it worth buying 100 dollars of Bitcoin?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Ultimately, it's up to you whether investing $100 in Bitcoin is worth it or not. If it's a one-time investment and you just want to try crypto out, we would recommend going with a lower amount since you can't profit much from $100 anyway.30.05.2022"
                ]
            }
        },
        "9": {
            "question": "Can Bitcoin be converted to cash?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "You can use a crypto exchange like Coinbase, Binance, Gemini or Kraken to turn Bitcoin into cash. This may be an easy method if you already use a centralized exchange and your crypto lives in a custodial wallet. Choose the coin and amount you'd like to sell, agree to the rates and your cash will be available to you.30.05.2022"
                ]
            }
        },
        "10": {
            "question": "Can I invest $5 in Bitcoin?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "New users get a $5 credit to invest in Bitcoin via the Bitcoin Investment Trust (GBTC), or any of the 1000+ stocks and ETFs we offer."
                ]
            }
        },
        "11": {
            "question": "How can I sell bitcoins for cash?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "To sell your bitcoin, you just need to go to an exchange and click to sell your bitcoin. Then you can choose how much you want to sell and move your cash to a bank. If you want to turn it straight to cash instead, then you should go through a bitcoin ATM.11.07.2022"
                ]
            }
        },
        "12": {
            "question": "Who gets the money when you buy Bitcoin?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "A buyer and seller agree on a price and a trade is executed over an exchange. So our $50k investor buys that amount of bitcoins and the seller receives the $50k in the form of a cash deposit. That seller may now keep it in the bank, buy other cryptos or withdraw it and spend it in any way they choose.27.11.2017"
                ]
            }
        },
        "13": {
            "question": "Is Bitcoin a good investment 2022?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Bitcoin's price has continued to fall through the first half of 2022, as has the stock market. Investors have turned to assets that perform well in a time of a slowing economy, higher inflation and rising interest rates.22.07.2022"
                ]
            }
        },
        "14": {
            "question": "Is Bitcoin still a good investment 2022?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Experts remain conflicted on whether bitcoin has bottomed out yet. Some say it already has, while others says bitcoin could fall as low as $10,000 in 2022. This volatility is a big part of why experts recommend keeping your crypto investments to less than 5% of your portfolio to begin with.22.07.2022"
                ]
            }
        }
    },
    "chicken": {
        "0": {
            "question": "Is chicken a dinosaur?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "So, are chickens dinosaurs? No \u2013 the birds are a distinct group of animals, but they did descend from the dinosaurs, and it's not too much of a twist of facts to call them modern dinosaurs. There are many similarities between the two types of animal, largely to do with bone structure."
                ]
            }
        },
        "1": {
            "question": "Is chicken a real animal?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "That seems at least partly true, but new research from Uppsala University now shows that the wild origins of the chicken are more complicated. The researchers mapped the genes that give most domesticated chickens yellow legs and found the genetic heredity derives from a closely related species, the gray jungle fowl.29.02.2008"
                ]
            }
        },
        "2": {
            "question": "What are the 3 types of chicken?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Chickens. Many different breeds of chickens have been developed for different purposes. For simplicity, you can place them into three general categories: Laying, meat-producing and dual-purpose breeds."
                ]
            }
        },
        "3": {
            "question": "What is small chicken called?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Miniature, or \u201cBantam,\u201d chickens are a subspecies of chicken that are much smaller than regular chickens. The average chicken weighs 5 to 10 pounds, whereas bantams only weigh 1 to 2.5 pounds! Most chicken breeds have a Bantam or miniature counterpart."
                ]
            }
        },
        "4": {
            "question": "Why are male chickens killed?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Male chicks are killed for two reasons: they cannot lay eggs and they are not suitable for chicken-meat production. This is because layer hens \u2014 and therefore their chicks \u2014 are a different breed of poultry to chickens that are bred and raised for meat production.22.09.2021"
                ]
            }
        },
        "5": {
            "question": "How big is a chicken brain?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "A chicken's brain is relatively small given the size of the head. It is the size of about 2 shelled peanuts, or a little bigger than a chicken's eyeball."
                ]
            }
        },
        "6": {
            "question": "Can chickens change gender?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "It is possible for a female chicken to take on external characteristics of a male, a phenomenon referred to as sex reversal."
                ]
            }
        },
        "7": {
            "question": "Is a hen a girl or boy?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Just as we humans are a particular species within mammals. And just as humans are divided into male and female and children and adults, chickens are likewise. In this context, a man equals a \u201crooster\u201d and a woman equals a \u201chen\u201d. Both are generally chickens, but the names indicate their gender and that they are adults.26.04.2019"
                ]
            }
        },
        "8": {
            "question": "Are there male chickens?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "The answer is yes, all male chickens are roosters. Roosters present a challenge to many backyard chicken owners because local jurisdictions usually have ordinances against roosters because they love to crow, which disturbs the neighbors.02.08.2022"
                ]
            }
        },
        "9": {
            "question": "Why are chickens called chooks?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "A domestic fowl; a chicken. Chook comes from British dialect chuck(y) 'a chicken; a fowl' which is a variant of chick. Chook is the common term for the live bird, although chook raffles, held in Australian clubs and pubs, have ready-to-cook chooks as prizes."
                ]
            }
        },
        "10": {
            "question": "What is a neutered chicken called?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "A capon is a male chicken that is gelded, or castrated, at a young age, and then fed a rich diet of milk or porridge. Larger than a chicken, a bit smaller than a turkey, but more flavorful than either, capons are full breasted with tender, juicy, flavorful meat that is well suited to roasting."
                ]
            }
        },
        "11": {
            "question": "Do dinosaurs Still Exist?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Other than birds, however, there is no scientific evidence that any dinosaurs, such as Tyrannosaurus, Velociraptor, Apatosaurus, Stegosaurus, or Triceratops, are still alive. These, and all other non-avian dinosaurs became extinct at least 65 million years ago at the end of the Cretaceous Period."
                ]
            }
        },
        "12": {
            "question": "What came first chicken or egg?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Back to our original question: with amniotic eggs showing up roughly 340 million or so years ago, and the first chickens evolving at around 58 thousand years ago at the earliest, it's a safe bet to say the egg came first. Eggs were around way before chickens even existed.29.03.2018"
                ]
            }
        },
        "13": {
            "question": "What was the first bird to fly?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Archaeopteryx: The Transitional Fossil. Paleontologists view Archaeopteryx as a transitional fossil between dinosaurs and modern birds. With its blend of avian and reptilian features, it was long viewed as the earliest known bird.14.03.2018"
                ]
            }
        },
        "14": {
            "question": "Can a chicken fly?",
            "answer": {
                "answerType": "text",
                "answer": [
                    "Chickens may have wings and fluffy feathers, but they're fairly dismal fliers, often going airborne for only a few yards before landing. The reason for their poor flight isn't as rhetorical as why they crossed the road.08.12.2016"
                ]
            }
        }
    }
}
Click to expand...

Hey, How can we upload JSON file to WP?
 
mmubeen8 said:
Want to import it as a post. i guess.
Click to expand...
Yeah, but there are some possibilities. For example importing a keyword as a post, getting questions and answers as Text in the article - or as meta data for the post. Or importing the question as article headline, the answer as text and using the keyword as taxenomy. That's than just a decission, how you want to build up your site, what architecture you might need and stuff like this ;)
 
light and bolts said:
Which code is the final version that works? (scrapes questions, answers and opens the toggles)
Click to expand...

For any reasons, I can't quote myself, so I'll just copy it =D
I wouldn't call it working, though - it's still rough and buggy, only a prototype reade for rewriting ;)
Python: 
from seleniumwire import webdriver
import random
import time
##################################################################
##################################################################
##################################################################
# Your Keyword List
kws = ["webscraping", "bitcoin", "chicken"]

# Limit scraped Questions ||| 0 == All Possible
limit = 15

##################################################################
##################################################################
##################################################################
##### FUNCTIONS
def clickyThingyXpath(xpath, wait, timeout, desc):
    startTime = time.time()
    time.sleep(wait)
    while True:
        checkTime = time.time() - timeout

        if startTime < checkTime:
            return(False)
        try:
            driver.find_element("xpath", xpath).click()
            return(True)
        except:
            time.sleep(1)
            pass

def findTexts(xpath, sleep):
    time.sleep(sleep)
    texts = list()
    _texts = driver.find_elements("xpath",xpath)
    for t in _texts:
        if len(t.text) > 5:
            texts.append(t.text)
    return(texts)

def findQuesionID(questions):
    _id = driver.find_element("xpath", '//*[text() = "'+questions+'"]/parent::*/parent::*/parent::*/parent::*/parent::*').get_attribute("id")
    return(_id)  

def XPathByQuestionID(_id):
    xpath = '//*[@id="'+_id+'"]/div/div/div[1]/div[4]'
    return(xpath)

def XPathByAnswerID(_id):
    clicky = '//*[@id="'+_id+'"]//*[@data-attrid="wa:/description"]'
    return(clicky)

def handleAnswerList(_id):
    _dict = dict()

    _dict["heading"] = findTexts('//*[@id="'+_id+'"]//*[@role="heading"]',0)
    findList = findTexts('//*[@id="'+_id+'"]//li',0)
    if len(findList) > 0:
        _dict["list"] = findList
        _dict["anwerType"] = "list"
        return(_dict)
    else:
        return(None)

def doWhateverYouWandwithThe(results):
    import json
    with open("results.json", "w") as jsonFile:
        json.dump(results, jsonFile, indent=4)

##################################################################
##################################################################
##################################################################
##### Script

results = dict()
for kw in kws:
    results[kw] = dict()
    ### Setting Up Webdriver
    ua = ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36","Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36"]
    options = webdriver.ChromeOptions()
    chrome_prefs = {}
    options.experimental_options["prefs"] = chrome_prefs
    chrome_prefs["profile.default_content_settings"] = {"images": 2}
    chrome_prefs["profile.managed_default_content_settings"] = {"images": 2}
    options.add_argument("--disable-web-security")
    options.add_argument("--disable-site-isolation-trials")
    options.add_argument("--disable-application-cache")
    options.add_experimental_option('excludeSwitches', ['enable-logging'])
    options.add_argument('--user-agent='+random.choice(ua))
    driver = webdriver.Chrome(executable_path=r"chromedriver.exe", chrome_options=options)

    ### Request
    url = 'https://www.google.com/search?q='+kw+'&start=0'
    driver.get(url)

    ### Click on Privacy Stuff - Optional
    clickyThingyXpath("/html/body/div[3]/div[3]/span/div/div/div/div[3]/div[1]/button[1]/div", 1, 60, "Handling Privacy Popup")

    ### Scraping
    seen = list()
    questions = findTexts("//*[@data-q]//div[@role='button']//div/span", 0)
    i = 0
    while True:
        for q in questions:
            try:
                # Preparing result dict
                results[kw][str(i)] = {"question":{}, "answer":{}}
                # check if question done allready, since in loop
                if q in seen:
                    continue
                # Get the question ID for further xpath usage
                qid = findQuesionID(q)
                print(kw+" ||| "+q)
                 # Open question
                clickyThingyXpath(XPathByQuestionID(qid), 0, 10, 'Clicking Question: "'+q+'"')
                # Scrape answer
                answer = findTexts(XPathByAnswerID(qid), 2)
                # check if answer is text only
                if len(answer) == 1:
                    anwerType = "Text"
                    while answer[0] == "":
                        answer = findTexts(XPathByAnswerID(qid), 2)
                    # Add to result dict
                    results[kw][str(i)]["question"] = q
                    results[kw][str(i)]["answer"]["answerType"] = "text"
                    results[kw][str(i)]["answer"]["answer"] = answer
                else:
                    # Not Text, so handle as list
                    answer = handleAnswerList(qid)
                    # Ignore if it's not a List as well
                    if answer == None:
                        seen.append(q)
                        continue
                    # Add to result dict
                    results[kw][str(i)]["question"] = q
                    results[kw][str(i)]["answer"] = answer
                seen.append(q)
                i+=1
                if i == limit:
                    break
            except:
                seen.append(q)
                continue
        if i == limit:
            break
        ### Did more questions pop up?
        _questions = findTexts("//*[@data-q]//div[@role='button']//div/span", 0)
        if len(questions) == len(_questions):
            break
        questions = _questions

    driver.close()

doWhateverYouWandwithThe(results)
 
light and bolts said:
Which code is the final version that works? (scrapes questions, answers and opens the toggles)
Click to expand...

Bugfisch said:
Should work with the actual version now. Uncomment line 97 if a European Cookie PopUp blocks the screen ;)

Code: 
from seleniumwire import webdriver
import random
import time
##################################################################
##################################################################
##################################################################
# Your Keyword List
kws = ["webscraping", "bitcoin", "chicken"]

# Limit scraped Questions ||| 0 == All Possible
limit = 15

##################################################################
##################################################################
##################################################################
##### FUNCTIONS
def clickyThingyXpath(xpath, wait, timeout, desc):
    startTime = time.time()
    time.sleep(wait)
    while True:
        checkTime = time.time() - timeout

        if startTime < checkTime:
            return(False)
        try:
            driver.find_element("xpath", xpath).click()
            return(True)
        except:
            time.sleep(1)
            pass

def findTexts(xpath, sleep):
    time.sleep(sleep)
    texts = list()
    _texts = driver.find_elements("xpath",xpath)
    for t in _texts:
        if len(t.text) > 5:
            texts.append(t.text)
    return(texts)

def findQuesionID(questions):
    _id = driver.find_element("xpath", '//*[text() = "'+questions+'"]/parent::*/parent::*/parent::*/parent::*/parent::*').get_attribute("id")
    return(_id)   

def XPathByQuestionID(_id):
    xpath = '//*[@id="'+_id+'"]/div/div/div[1]/div[4]'
    return(xpath)

def XPathByAnswerID(_id):
    clicky = '//*[@id="'+_id+'"]//*[@data-attrid="wa:/description"]'
    return(clicky)

def handleAnswerList(_id):
    _dict = dict()

    _dict["heading"] = findTexts('//*[@id="'+_id+'"]//*[@role="heading"]',0)
    findList = findTexts('//*[@id="'+_id+'"]//li',0)
    if len(findList) > 0:
        _dict["list"] = findList
        _dict["anwerType"] = "list"
        return(_dict)
    else:
        return(None)

def doWhateverYouWandwithThe(results):
    import json
    with open("results.json", "w") as jsonFile:
        json.dump(results, jsonFile, indent=4)

##################################################################
##################################################################
##################################################################
##### Script

results = dict()
for kw in kws:
    results[kw] = dict()
    ### Setting Up Webdriver
    ua = ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36","Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36"]
    options = webdriver.ChromeOptions()
    chrome_prefs = {}
    options.experimental_options["prefs"] = chrome_prefs
    chrome_prefs["profile.default_content_settings"] = {"images": 2}
    chrome_prefs["profile.managed_default_content_settings"] = {"images": 2}
    options.add_argument("--disable-web-security")
    options.add_argument("--disable-site-isolation-trials")
    options.add_argument("--disable-application-cache")
    options.add_experimental_option('excludeSwitches', ['enable-logging'])
    options.add_argument('--user-agent='+random.choice(ua))
    driver = webdriver.Chrome(executable_path=r"chromedriver.exe", chrome_options=options)

    ### Request
    url = 'https://www.google.com/search?q='+kw+'&start=0'
    driver.get(url)

    ### Click on Privacy Stuff - Optional
    # clickyThingyXpath("/html/body/div[3]/div[3]/span/div/div/div/div[3]/div[1]/button[1]/div", 1, 60, "Handling Privacy Popup")

    ### Scraping
    seen = list()
    questions = findTexts("//*[@data-q]//div[@role='button']//div/span", 0)
    i = 0
    while True:
        for q in questions:
            try:
                # Preparing result dict
                results[kw][str(i)] = {"question":{}, "answer":{}}
                # check if question done allready, since in loop
                if q in seen:
                    continue
                # Get the question ID for further xpath usage
                qid = findQuesionID(q)
                print(kw+" ||| "+q)
                 # Open question
                clickyThingyXpath(XPathByQuestionID(qid), 0, 10, 'Clicking Question: "'+q+'"')
                # Scrape answer
                answer = findTexts(XPathByAnswerID(qid), 2)
                # check if answer is text only
                if len(answer) == 1:
                    anwerType = "Text"
                    while answer[0] == "":
                        answer = findTexts(XPathByAnswerID(qid), 2)
                    # Add to result dict
                    results[kw][str(i)]["question"] = q
                    results[kw][str(i)]["answer"]["answerType"] = "text"
                    results[kw][str(i)]["answer"]["answer"] = answer
                else:
                    # Not Text, so handle as list
                    answer = handleAnswerList(qid)
                    # Ignore if it's not a List as well
                    if answer == None:
                        seen.append(q)
                        continue
                    # Add to result dict
                    results[kw][str(i)]["question"] = q
                    results[kw][str(i)]["answer"] = answer
                seen.append(q)
                i+=1
                if i == limit:
                    break
            except:
                seen.append(q)
                continue
        if i == limit:
            break
        ### Did more questions pop up?
        _questions = findTexts("//*[@data-q]//div[@role='button']//div/span", 0)
        if len(questions) == len(_questions):
            break
        questions = _questions

    driver.close()

doWhateverYouWandwithThe(results)
Click to expand...

This one. It will create the JSON file for results.
 
welshman said:
Anyway to output to csv instead of json file?
Click to expand...
try this . write your folder path to (draft )

# save current keyword as a csv in draft folder
df.to_csv(f'{sys.path[0]}\draft\{excel}.csv', index=False)

print("\n**********************************")
print(f'saving ({excel}) to csv - finished')
print("**********************************")
 
desperado596 said:
try this . write your folder path to (draft )

# save current keyword as a csv in draft folder
df.to_csv(f'{sys.path[0]}\draft\{excel}.csv', index=False)

print("\n**********************************")
print(f'saving ({excel}) to csv - finished')
print("**********************************")
Click to expand...
thanks, but I get this error:

df.to_csv(f'{sys.path[0]}C:\Users\xboxb\Downloads\PAA\draft\{excel}.csv', index=False)
^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape
 
import sys
df.to_csv(f'{sys.path[0]}\C:\Users\xboxb\Downloads\PAA\draft\{excel}.csv', index=False)

try again please.
 
df.to_csv(f'{sys.path[0]}\C:\Users\xboxb\Downloads\PAA\draft\{excel}.csv', index=False)
^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 3-4: truncated \UXXXXXXXX escape
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top