[Please HELP] How to scrape article URLs from the sitemap by category?

[Almostbr0ke]

Hello,
today I found out that two of my competitors are getting over 100k monthly visitors from different search engines than Google (local). And to be honest, the content on both domains is not great and what is even worse is domain authority, I don't specifically mean DA, but the link profile is very weak, with almost no backlinking and even on-page stuff like images or interlinks, these sites has almost none of these.. My question to you is, how can I scrape hURLs by category so I can rip the whole site?
One site has a rank math sitemap, and the other one has yoast SEO sitemap.
I would really appreciate your help here.

 

[sda]

I would suggest a python script. There are a lot of tutorials in the Internet. And in todays developing world, even AI writes scripts very well.

 

[Almostbr0ke]

I would suggest a python script. There are a lot of tutorials in the Internet. And in todays developing world, even AI writes scripts very well.

Would I need proxies for that?

 

[sda]

You will only need proxies, if the target website will start blocking your IP.

 

[Almostbr0ke]

You will only need proxies, if the target website will start blocking your IP.

Thank you, I will try to create something for this with chatgpt

 

[prodon]

You can use sitemap explorer chrome extension

 

[Almostbr0ke]

You can use sitemap explorer chrome extension

Thanks, I will try that first, haha

 

[Comsign]

Hi, try this method to scrape article URLs from a sitemap by category, follow these steps:

1. Locate the sitemap (usually in XML format) on the website, often found at website.com/sitemap.xml.
2. Use a script or a tool that can parse XML. Python with libraries like BeautifulSoup or lxml is a good choice.
3. Load the sitemap into your script.
4. Filter the URLs by the desired category, which may be indicated in the URL structure or within specific XML tags.
5. Extract and save the filtered URLs.

Remember to respect the website's robots.txt file and scraping policies, also remember about proxies.

 

[Almostbr0ke]

Hi, try this method to scrape article URLs from a sitemap by category, follow these steps:

1. Locate the sitemap (usually in XML format) on the website, often found at website.com/sitemap.xml.
2. Use a script or a tool that can parse XML. Python with libraries like BeautifulSoup or lxml is a good choice.
3. Load the sitemap into your script.
4. Filter the URLs by the desired category, which may be indicated in the URL structure or within specific XML tags.
5. Extract and save the filtered URLs.

Remember to respect the website's robots.txt file and scraping policies, also remember about proxies.

Unfortunately the categories are not with in the url

 

[bottingisfine]

did you find a good solution for this?

 

[Almostbr0ke]

did you find a good solution for this?

Not really, I have not found an easy way to categorize urls from the sitemap file

 

[leidi]

You will only need proxies if the target website will start blocking your IP.

 

[Almostbr0ke]

You will only need proxies if the target website will start blocking your IP.

But how can I scrape it by category, if the category is not in the url? The sitemap is in this format: domain.com/article-title

 

[goodbye tobey]

But how can I scrape it by category, if the category is not in the url? The sitemap is in this format: domain.com/article-title

in this case you have to access every url and scrape the category from from the article's page (assuming the category is provided in the article)