Need help with selenium, headless browser and extension

[Floccer]

Hi

I have been trying to build headless scraper with extension. Scraping works well if browser is not in headless mode. Both browsers are set with profile that has the extension installed. I could ditch the extension if elements wouldn't have dynamic variables. I have been unable to identify right element to get data from the site.

Chromedriver: With chrome it seems that extensions are not supported in headless mode, at least haven't found a way.
Geckodriver: Extensions work with headless but now the text is placed on clipboard and it seems clipboard is not shared from headless mode to host machine?

Could there be some other solutions in windows to get scraping working in headless mode?

 

[AutopilotMoney]

Could there be some other solutions in windows to get scraping working in headless mode?

Pure HTTP requests with PHP, Python, ...

Why do you need the browser emulation?

 

[Floccer]

Pure HTTP requests with PHP, Python, ...

Why do you need the browser emulation?

i'm not skilled enough to get the data as the element names changes and i always get wrong results from the page.

I found browser extension that works every time and it will give me exactly what i want from the page. Well i try again with the requests as i know that would be much cleaner solution to this.

 

[Smackhead]

why does the browser need to be in headless mode? I usually use selenium if I need to scrape some data. If you want to send me some details I might be able to help.

 

[Floccer]

why does the browser need to be in headless mode? I usually use selenium if I need to scrape some data. If you want to send me some details I might be able to help.

I use it on my computer that i use regularly for other work and its bothering me when the browser popups. Also i was hoping to be able to somehow run it isolated from the clipboard. If i or script copies text it can mess things up. I have just been leaving that running at night so it won't bother me then.

 

[Smackhead]

I use it on my computer that i use regularly for other work and its bothering me when the browser popups. Also i was hoping to be able to somehow run it isolated from the clipboard. If i or script copies text it can mess things up. I have just been leaving that running at night so it won't bother me then.

ah ok I see. So you can run selenium headless what are you using the extension for?

 

[Floccer]

ah ok I see. So you can run selenium headless what are you using the extension for?

The extension will extract data from serp as i'm unable to do it with find_elements. There seems to be so many dynamic variables that i don't know how i could locate right elements.
I know its possible with xpath axes like following-sibling, descendant etc. Those are too advanced stuff for me to understand and haven't had time to learn those.

 

[Deleted member 977120]

Extensions in Chrome + Selenium don't work with headless. I'd say don't go with headless?

 

[Floccer]

Extensions in Chrome + Selenium don't work with headless. I'd say don't go with headless?

seems to be only option if i stick with extension

 

[Smackhead]

The extension will extract data from serp as i'm unable to do it with find_elements. There seems to be so many dynamic variables that i don't know how i could locate right elements.
I know its possible with xpath axes like following-sibling, descendant etc. Those are too advanced stuff for me to understand and haven't had time to learn those.

If you want to DM me some examples I might be able to help.

 

[cutty]

Pure HTTP requests with PHP, Python, ...

Why do you need the browser emulation?

Replace selenium with requests, and learn what you need to learn to accomplish your task.

 

[cutty]

The extension will extract data from serp as i'm unable to do it with find_elements. There seems to be so many dynamic variables that i don't know how i could locate right elements.
I know its possible with xpath axes like following-sibling, descendant etc. Those are too advanced stuff for me to understand and haven't had time to learn those.

There is always a way, dear.

 

[UncomfortablyNumb]

The extension will extract data from serp as i'm unable to do it with find_elements. There seems to be so many dynamic variables that i don't know how i could locate right elements.
I know its possible with xpath axes like following-sibling, descendant etc. Those are too advanced stuff for me to understand and haven't had time to learn those.

I'd really advise getting to grips with XPath, it's going to be the best solution for what you want to do & will be a useful skill for any future projects.