How to Identify Article URLs on WordPress Homepages?

[hercai]

Hello!
When I visit the homepage of a WordPress site, how can I identify the URLs of the articles on the site? My goal is to capture the URL of any article correctly and proceed with further actions. I’ve tried the following methods, but they weren’t successful: ".post-title a", "h1/2/3/4/5.entry-title a" and "article a"

Which parameter, element, or XPath method should I use to detect article URLs properly? Can you help me with this?

 

[tazarbm]

if it's not a big site you can pull the site's sitemap (whateversite.com/sitemap.xml) and get the URLs from there...

If it's a big site... hmm, you can probably use site auditing software like Screaming Frog (or any other SEO auditing tool) that can find all URLs, but then you'll have to figure out a way to extract / copy / whatever you want to do with them...

You can also use Scrapebox, SB is amazing (and still the best tool on the market) for any task that involves scraping, gathering, harvesting, sorting, trimming, cleaning anything about URLs, keywords, footprints, etc. And I think you can do this with one of the free add-ons, you don't need the paid plugins for this, but I don't remember cause I've not used SB in 2 years.... But I think it's one of the free addons, you just have to buy the software and proxies...

 

[hercai]

if it's not a big site you can pull the site's sitemap (whateversite.com/sitemap.xml) and get the URLs from there...

If it's a big site... hmm, you can probably use site auditing software like Screaming Frog (or any other SEO auditing tool) that can find all URLs, but then you'll have to figure out a way to extract / copy / whatever you want to do with them...

You can also use Scrapebox, SB is amazing (and still the best tool on the market) for any task that involves scraping, gathering, harvesting, sorting, trimming, cleaning anything about URLs, keywords, footprints, etc. And I think you can do this with one of the free add-ons, you don't need the paid plugins for this, but I don't remember cause I've not used SB in 2 years.... But I think it's one of the free addons, you just have to buy the software and proxies...

My goal is to visit the homepage of a site built on WordPress and then navigate to any article on the site. I'll be doing this in Python. This way, I’ll first visit the homepage and then proceed to an article page. However, I'm having trouble identifying the articles. I need to use element or xpath methods in Python to determine which of the titles on the homepage belong to an article.

 

[tazarbm]

My goal is to visit the homepage of a site built on WordPress and then navigate to any article on the site. I'll be doing this in Python. This way, I’ll first visit the homepage and then proceed to an article page. However, I'm having trouble identifying the articles. I need to use element or xpath methods in Python to determine which of the titles on the homepage belong to an article.

oh, I don't know Python, so I can't help. Sorry!

 

[hercai]

oh, I don't know Python, so I can't help. Sorry!

I'll handle the Python part myself. I think it would be best if I do some more research. I'll try to find a small code snippet that can help me detect the article. Thank you very much for your attention and assistance.

 

[tazarbm]

I'll handle the Python part myself. I think it would be best if I do some more research. I'll try to find a small code snippet that can help me detect the article. Thank you very much for your attention and assistance.

no problem, I didn't do anything anyway

Good luck figuring it out!

 

[DennisParsens]

There might be a simple solution if the URL's/directory paths of the articles are following a consistent structure:

Say you have domain .com/posts/article_title_1. You could scrape from the page all the URL's found inside the /posts/ directory, hope you get the idea.