[CODE] People Also Ask Scraping with Python [Upgraded code from iam_ironman]

MasonF said:

not all pages will return 3 questions so

Python:

import requests
from lxml.html import fromstring
import lxml.html

file1 = open('YourPath/PeopleAlsoAsk/keywords.txt', 'r')
file2 = open('YourPath/PeopleAlsoAsk/paa.txt', 'w')
Lines = file1.readlines()
header = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36',
        'Accept-Language': 'tr-tr,en;q=0.9',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
        }
for line in Lines:
        query = line.strip()
        response = requests.get(f'https://www.google.com/search?q={query}&start=0', headers=header).text
        tree = lxml.html.fromstring(response)
        nodes = tree.xpath('//@data-q')
        print(query)
        file2.write(query)
        file2.write(": ")
        for node in nodes:
            file2.write(node)
            file2.write(" , ")
        file2.write("\n")
file1.close()
file2.close()

Click to expand...

MasonF said:

not all pages will return 3 questions so

Python:

import requests
from lxml.html import fromstring
import lxml.html

file1 = open('YourPath/PeopleAlsoAsk/keywords.txt', 'r')
file2 = open('YourPath/PeopleAlsoAsk/paa.txt', 'w')
Lines = file1.readlines()
header = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36',
        'Accept-Language': 'tr-tr,en;q=0.9',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
        }
for line in Lines:
        query = line.strip()
        response = requests.get(f'https://www.google.com/search?q={query}&start=0', headers=header).text
        tree = lxml.html.fromstring(response)
        nodes = tree.xpath('//@data-q')
        print(query)
        file2.write(query)
        file2.write(": ")
        for node in nodes:
            file2.write(node)
            file2.write(" , ")
        file2.write("\n")
file1.close()
file2.close()

Click to expand...

Madoxx said:

fUntastic!

This will collect all of the questions, in this case, whether 1 or 5, right?

Click to expand...

MasonF said:

correct

Click to expand...

Madoxx said:

Is there a way to somehow collect beyond what's shown?
When I google a keyword, it shows me 4 PAAs, and when I click on the last one, it shows me two more. How can we simulate click with python to scrape additional PAAs?

Click to expand...

Ramazan said:

I'm looking for too. Need 10 question. Probably selenium is the answer.

Click to expand...

Madoxx said:

Lemme know if you figure out something...

If you want a table view of the results, you can:
1. change the format of the export file from .txt to .csv
2. after each result (node) write comma , without spaces

After that, you can use converter to get XLSX file which you can view in your Google Sheets

BTW. where can I find a list of available languages for PPA, or is it only for English?

Click to expand...

Madoxx said:

Is there a way to somehow collect beyond what's shown?
When I google a keyword, it shows me 4 PAAs, and when I click on the last one, it shows me two more. How can we simulate click with python to scrape additional PAAs?

Click to expand...

Ramazan said:

I'm too lazy for xlsx file thats why Its txt lul

If i find something i will share.

Click to expand...

Rufin jazz said:

This is interesting. Thank you deeply to all contributors on this thread.

Click to expand...

Ramazan said:

I'm looking for too. Need 10 question. Probably selenium is the answer.

Click to expand...