Need help with selenium, headless browser and extension

Floccer

Floccer

Regular Member
Joined
Mar 2, 2019
Messages
365
Reaction score
163
Hi

I have been trying to build headless scraper with extension. Scraping works well if browser is not in headless mode. Both browsers are set with profile that has the extension installed. I could ditch the extension if elements wouldn't have dynamic variables. I have been unable to identify right element to get data from the site.

Chromedriver: With chrome it seems that extensions are not supported in headless mode, at least haven't found a way.
Geckodriver: Extensions work with headless but now the text is placed on clipboard and it seems clipboard is not shared from headless mode to host machine?

Could there be some other solutions in windows to get scraping working in headless mode?
 
Advertise on BHW
AutopilotMoney said:
Pure HTTP requests with PHP, Python, ...

Why do you need the browser emulation?
Click to expand...
i'm not skilled enough to get the data as the element names changes and i always get wrong results from the page.

I found browser extension that works every time and it will give me exactly what i want from the page. Well i try again with the requests as i know that would be much cleaner solution to this.
 
why does the browser need to be in headless mode? I usually use selenium if I need to scrape some data. If you want to send me some details I might be able to help.
 
Smackhead said:
why does the browser need to be in headless mode? I usually use selenium if I need to scrape some data. If you want to send me some details I might be able to help.
Click to expand...
I use it on my computer that i use regularly for other work and its bothering me when the browser popups. Also i was hoping to be able to somehow run it isolated from the clipboard. If i or script copies text it can mess things up. I have just been leaving that running at night so it won't bother me then.
 
Floccer said:
I use it on my computer that i use regularly for other work and its bothering me when the browser popups. Also i was hoping to be able to somehow run it isolated from the clipboard. If i or script copies text it can mess things up. I have just been leaving that running at night so it won't bother me then.
Click to expand...
ah ok I see. So you can run selenium headless what are you using the extension for?
 
Smackhead said:
ah ok I see. So you can run selenium headless what are you using the extension for?
Click to expand...
The extension will extract data from serp as i'm unable to do it with find_elements. There seems to be so many dynamic variables that i don't know how i could locate right elements.
I know its possible with xpath axes like following-sibling, descendant etc. Those are too advanced stuff for me to understand and haven't had time to learn those.
 
Extensions in Chrome + Selenium don't work with headless. I'd say don't go with headless?
 
Unity7 said:
Extensions in Chrome + Selenium don't work with headless. I'd say don't go with headless?
Click to expand...
seems to be only option if i stick with extension
 
Floccer said:
The extension will extract data from serp as i'm unable to do it with find_elements. There seems to be so many dynamic variables that i don't know how i could locate right elements.
I know its possible with xpath axes like following-sibling, descendant etc. Those are too advanced stuff for me to understand and haven't had time to learn those.
Click to expand...
If you want to DM me some examples I might be able to help.
 
Floccer said:
The extension will extract data from serp as i'm unable to do it with find_elements. There seems to be so many dynamic variables that i don't know how i could locate right elements.
I know its possible with xpath axes like following-sibling, descendant etc. Those are too advanced stuff for me to understand and haven't had time to learn those.
Click to expand...
There is always a way, dear.
 
Floccer said:
The extension will extract data from serp as i'm unable to do it with find_elements. There seems to be so many dynamic variables that i don't know how i could locate right elements.
I know its possible with xpath axes like following-sibling, descendant etc. Those are too advanced stuff for me to understand and haven't had time to learn those.
Click to expand...

I'd really advise getting to grips with XPath, it's going to be the best solution for what you want to do & will be a useful skill for any future projects.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top