XPath selectors

Pieter uit de Betuwe

Pieter uit de Betuwe

Newbie
Joined
Apr 3, 2026
Messages
1
Reaction score
0
Hi @lordgl
For 'complex' XPath settings I use AI as help.
I use Text.Cortex for this. Mostly Claude Sonnet (now 4.6) with files stored in the Knowledge section.
Copy Paste the source code and ask first what the XPath is to extract ....
So step by step.
Then let it combine all XPath rules into 1 rule.

I made an XPath to extract LinkedIn companies and Brave.com search results.

Where are you from? The Netherlands also?
lordgl said:
Hey welkom! Nice to see someone from the Netherlands here, based in the north. Interesting that you combine Scrapebox with Screaming Frog, do you find that your XPath selectors break a lot when sites change their layout? Thats always been the most annoying part of scraping for me
Click to expand...
 
Advertise on BHW
Pieter uit de Betuwe said:
Hi @lordgl
For 'complex' XPath settings I use AI as help.
I use Text.Cortex for this. Mostly Claude Sonnet (now 4.6) with files stored in the Knowledge section.
Copy Paste the source code and ask first what the XPath is to extract ....
So step by step.
Then let it combine all XPath rules into 1 rule.

I made an XPath to extract LinkedIn companies and Brave.com search results.

Where are you from? The Netherlands also?
Click to expand...
Thats a smart workflow ;) LinkedIn is especially tricky with their DOM changes.
Funny enough we're actually building a platform right now that deals with exactly this problem. When selectors break, the system automatically tries a cascade of fallback approaches, cached selectors first, then aria/role attributes, then structural patterns. And if everything fails it uses AI vision to re-map the selectors automatically. Still in development but the core engine is working.

Have you thought about automating that part of your workflow too or do you prefer the manual AI-assisted approach?
BTW i am from Leeuwarden :)
 
You can exactly get xpath easily by clicking on inspect element, select the part, copy xpath
 
The fallback cascade you described is solid. One thing worth adding at the caching stage: prioritize attribute-based paths like [@data-testid] or [@aria-label] over positional ones like div[3]/span[2] — positional XPath breaks on any layout change, attribute-based paths survive most redesigns. For LinkedIn specifically, their class names are obfuscated hashes so class-based selectors are a dead end from day one; you have to go semantic or structural.
 
Honestly, relying on AI to build and maintain those complex xpath rules is a total game changer for keeping your scrapers from breaking every time a site updates. It definitely saves you so much manual headache and keeps the workflow way more efficient
 
vierasen said:
The fallback cascade you described is solid. One thing worth adding at the caching stage: prioritize attribute-based paths like [@data-testid] or [@aria-label] over positional ones like div[3]/span[2] — positional XPath breaks on any layout change, attribute-based paths survive most redesigns. For LinkedIn specifically, their class names are obfuscated hashes so class-based selectors are a dead end from day one; you have to go semantic or structural.
Click to expand...
Spot on about attribute-based paths, thats exactly the priority order we use. data-testid and aria-label first, positional as last resort. LinkedIn is a good example, their class names are basically random hashes that change every deploy. We also found that role-based selectors (like [role='button']) are surprisingly stable across redesigns, even more than aria-labels sometimes. The tricky part is when sites remove semantic attributes entirely, thats where the vision fallback becomes useful as a last resort
 
Good point on role-based selectors — we haven't weighted them separately yet, been treating them as part of the same aria bucket. Might be worth splitting that into its own tier. And yeah, the "no semantic attributes at all" case is the nightmare scenario. Vision fallback is the only real option there, but it's expensive. Do you cache the vision-mapped selectors per page version/hash, or re-run vision every time it fails?
 
vierasen said:
Good point on role-based selectors — we haven't weighted them separately yet, been treating them as part of the same aria bucket. Might be worth splitting that into its own tier. And yeah, the "no semantic attributes at all" case is the nightmare scenario. Vision fallback is the only real option there, but it's expensive. Do you cache the vision-mapped selectors per page version/hash, or re-run vision every time it fails?
Click to expand...
Yeah vision is way too expensive to run every time, we cache aggressively. When vision maps a new selector it gets stored and becomes the cached first-try for that intent. We also version it loosely, if the cached selector starts failing (5+ misses in a short window) it triggers a re-inspection. So vision runs maybe once every few weeks per page, not on every action. The expensive part was getting the circuit breaker right so it doesnt burn through API credits when a page is genuinely broken vs just temporarily glitchy
 
The circuit breaker distinction is the hard part — curious how you differentiate "page is broken" from "temporarily glitchy". Do you look at HTTP status codes, compare DOM structure against a baseline snapshot, or just rely purely on the miss-count window? We've been thinking about something similar but haven't landed on a clean signal yet. Also 5 misses as the threshold — is that per session or across distributed runs?
 
vierasen said:
The circuit breaker distinction is the hard part — curious how you differentiate "page is broken" from "temporarily glitchy". Do you look at HTTP status codes, compare DOM structure against a baseline snapshot, or just rely purely on the miss-count window? We've been thinking about something similar but haven't landed on a clean signal yet. Also 5 misses as the threshold — is that per session or across distributed runs?
Click to expand...
Good question honestly, cleanly distinguishing temporary glitches from real layout changes is something we're still refining. Short version: we look at failure patterns across multiple accounts rather than just one, if a single account misses a selector its probably a page load issue, if several accounts fail on the same intent within a short window thats likely a real change. Plus basic checks like whether the page even loaded properly before blaming the selector. Theres definitely more we could do there with DOM diffing and baseline snapshots but the simple approach has been working well enough so far.

Actually, we're getting pretty close to opening a beta for the platform. Would you be interested in testing it when we do? Given your experience with scraping and selector challenges I think you'd get a lot out of it and your feedback would be super valuable for us :)
 
Yep, AI helps, but XPath selectors still break. I keep it simple: build a base path that grabs the element, then layer in stable attributes first (data-testid, aria-label), with a last resort on structural hints. Add a lightweight fallback chain and cache working selectors so you don’t re-run maps every time. If you want, I can post a minimal example.
 
Honestly, using AI to generate those dynamic xpaths is such a game changer for keeping your data clean without the manual headache. Just lean into the AI workflow and you'll spend way less time babysitting your scrapers whenever those pesky site layouts shift
 
Im using claude
I let it use my browser and extract the xpath for me or sometimes i use Css selector
 
CTRPhantom said:
Yep, AI helps, but XPath selectors still break. I keep it simple: build a base path that grabs the element, then layer in stable attributes first (data-testid, aria-label), with a last resort on structural hints. Add a lightweight fallback chain and cache working selectors so you don’t re-run maps every time. If you want, I can post a minimal example.
Click to expand...
Please post a minimal example
 
XPath selectors are useful for scraping and automation tasks.
They help locate elements more precisely compared to basic selectors.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top