Is scraping google people also ask viable in pure html requests or is the garbled javascript making it too cumbersome?

RowdySadjit

Banned - Selling via PM
Joined
Dec 20, 2021
Messages
275
Reaction score
115
I will have a look myself but wondered if anyone else had thought about/tried it.

I always try to go for requests as the first port of call and move back to browser if it seems too impractical.
 
There are already FOSS repos that take care of that, I used one of those a year ago and they work quite well.
 
It's easier with selenium. Here's a piece of code to get an idea, just made one for myself a while back.

Python:
        for elem in elements:
            question = elem.find_element_by_class_name('wWOJcd')

            try:
                answer = elem.find_element_by_class_name('ILfuVd')
                answerh = answer.get_attribute('innerHTML')
            except Exception:# pylint: disable=broad-except
                try:
                    answer = elem.find_element_by_class_name('L2AgXb')
                    yttag = answer.find_element_by_tag_name('a')
                    answerh = yttag.get_attribute("href")
                except Exception:# pylint: disable=broad-except
                    answerh = None


            if answerh is not None:
                with open(self.filename,"a", encoding="utf-8") as publishfile:
                    publishfile.write(f"<h2>{question.text}</h2><p>{answerh}</p>")
 
Of course it is, just like everything else. That goes without saying. Easiest is not always best. Pure html requests are like 1000x more efficient (in terms of less resource use) and just way cooler.
 
Back
Top