PYTHON - Unload part of DOM of a page while scrapping PAA results (firefox & selenium)

Hey there !

Today I faced a big problem ! I make a software that work perfectly with my big computer but hurt when launching it in a Windows VPS.

Its a Python Bot that scrap PAA results in Google for a given word. Its really simple and you can test this at home if you want: when you click on a People Also Ask question it unspoil the body text and it make news PAA appear. The fact is that you can sometimes do it 1000 times for a word and with a not powerfull computer the navigator struggle to keep rendering the page (with Chrome it was crashing in my VPS).

I goes with firefox and works very well, no crash, but when reach a certain level of "DOM size" it begin go very slowly and collect a new PAA each 2 hours instead of 1 second..

Anyone faced a similar problem or got an idea ? I was planning to do something like: when scrap a new PAA, remove the DOM of everything that go before it.

But not very sure if that is a good solution..

Hope someone can help me lol, thanks for reading me guys !

Bisous

Removing the dom elements I am not sure will it solve your issue.
For this, I am using puppeteer and you can inject Jquery and use $(elements).remove() to remove them,

Why are you clicking 1,000 times anyway?
You click once, and move to the next.

You should not get past 40 clicks/query.
Also, selenium is not thread friendly, and will eat up resources like crazy after a certain point.

You should close the driver after 50-100 queries, and restart it.

WhiteHatCoder said:

Removing the dom elements I am not sure will it solve your issue.
For this, I am using puppeteer and you can inject Jquery and use $(elements).remove() to remove them,

Click to expand...

Oh yes nice idea thanks you !

Alexion said:

Why are you clicking 1,000 times anyway?
You click once, and move to the next.

You should not get past 40 clicks/query.
Also, selenium is not thread friendly, and will eat up resources like crazy after a certain point.

You should close the driver after 50-100 queries, and restart it.

Click to expand...

I am clicking 1000 times because for scrapping the PAA I need to open them, I mean dont go to the link but click to unspoil it ANNNNNND for make more PAA appears ! If I dont click I just scrapp the 5/6 PAA by words and nothing more, if I open each one I get PAA infinite

The firefox crash not because of an action but because the result is a google search result page ultraaabiggggg with lot of PAA load inside !! Just look how big come a result page for just one sentence !

Kierkegaard said:

Oh yes nice idea thanks you !



I am clicking 1000 times because for scrapping the PAA I need to open them, I mean dont go to the link but click to unspoil it ANNNNNND for make more PAA appears ! If I dont click I just scrapp the 5/6 PAA by words and nothing more, if I open each one I get PAA infinite

The firefox crash not because of an action but because the result is a google search result page ultraaabiggggg with lot of PAA load inside !! Just look how big come a result page for just one sentence !

View attachment 228545

Click to expand...

You missed my point.

How many question/answer pairs are you aiming at for one query?
You shouldn't go past 40, or even 50 anyway, it's enough.

You only need to click once per question.
That's about 50 clicks at max per query.

Once your query is finished, you load google.com/search again, and the DOM goes away.

Alexion said:

You missed my point.

How many question/answer pairs are you aiming at for one query?
You shouldn't go past 40, or even 50 anyway, it's enough.

You only need to click once per question.
That's about 50 clicks at max per query.

Once your query is finished, you load google.com/search again, and the DOM goes away.

Click to expand...

Im aiming the maximum PAA by query because I dont want to waste those.
Im sure you better for this than me but I dont understand why you told me to dont overcome 40 or 50 PAA by query ? Why it is enough ? My bot create blog posts with those so I need a maximum

Yes I just click once per question, not two or three time, dont know why you tell me this


Yes exactly what I doo but when with a query I got like 100+ PAA it comes to crash my web browser. Sure I can limit it to dont overcome my VPS power but... if I want scrapp more

I did those with Chromium with 200/300 PAA and no memory issue.

But @Alexion is Right, after a certain point QA is pretty much irrelevant to the query subject, so
maybe a better solution is to have 1k variations with some common how/why etc words before and
after so results are more relevant.

It would be better if you attach the code with which the problem arises, then you will be helped faster.

Surfinite said:

It would be better if you attach the code with which the problem arises, then you will be helped faster.

Click to expand...

No console error, Im developer I can solve this kind of probleme. What I tell here is that my nav dont crash but go slower and slower each PAA it load

Did you try anything? And why don't u like the solution to remove DOM elements? Its kinda obvious, after clicking the PPA for 20 times, the HTML for those elements in my case contains 400k chars, which is 50% of the whole page HTML. So the content (and load on your cpu and ram) scales linearly with the number of clicks made. So its expected that any harware will go slower and slower with each PAA load.
Removing elements worth a try 100%. But if I were so stubborn, there is some space for freestyling:
When you click a PAA, it makes a request to /async/callback/...<data-bs>..., and the response contains new PAAs (with their corresponding <data-bs>) that JS then injects into page html.
You can use Chrome DevTools Protocol to intercept those requests, extract new data-bs, fail them, so that JS does not populate page by itself. Then you would either try to make these requests yourself from code (I doubt its worth it) or replace existing PAAs on the page with the new data-bs. That way you would keep only first 4-5 PAAs rendered on the page, but your request interceptor would gather them all.