[Please HELP] How to scrape article URLs from the sitemap by category?

A

Almostbr0ke

BANNED
Joined
Jan 11, 2021
Messages
1,821
Reaction score
717
Hello,
today I found out that two of my competitors are getting over 100k monthly visitors from different search engines than Google (local). And to be honest, the content on both domains is not great and what is even worse is domain authority, I don't specifically mean DA, but the link profile is very weak, with almost no backlinking and even on-page stuff like images or interlinks, these sites has almost none of these.. My question to you is, how can I scrape hURLs by category so I can rip the whole site?
One site has a rank math sitemap, and the other one has yoast SEO sitemap.
I would really appreciate your help here.
 
Advertise on BHW
I would suggest a python script. There are a lot of tutorials in the Internet. And in todays developing world, even AI writes scripts very well.
 
sda said:
I would suggest a python script. There are a lot of tutorials in the Internet. And in todays developing world, even AI writes scripts very well.
Click to expand...
Would I need proxies for that?
 
You will only need proxies, if the target website will start blocking your IP.
 
Hi, try this method to scrape article URLs from a sitemap by category, follow these steps:
  1. Locate the sitemap (usually in XML format) on the website, often found at website.com/sitemap.xml.
  2. Use a script or a tool that can parse XML. Python with libraries like BeautifulSoup or lxml is a good choice.
  3. Load the sitemap into your script.
  4. Filter the URLs by the desired category, which may be indicated in the URL structure or within specific XML tags.
  5. Extract and save the filtered URLs.
Remember to respect the website's robots.txt file and scraping policies, also remember about proxies.
 
Comsign said:
Hi, try this method to scrape article URLs from a sitemap by category, follow these steps:
  1. Locate the sitemap (usually in XML format) on the website, often found at website.com/sitemap.xml.
  2. Use a script or a tool that can parse XML. Python with libraries like BeautifulSoup or lxml is a good choice.
  3. Load the sitemap into your script.
  4. Filter the URLs by the desired category, which may be indicated in the URL structure or within specific XML tags.
  5. Extract and save the filtered URLs.
Remember to respect the website's robots.txt file and scraping policies, also remember about proxies.
Click to expand...
Unfortunately the categories are not with in the url
 
You will only need proxies if the target website will start blocking your IP.
 
leidi said:
You will only need proxies if the target website will start blocking your IP.
Click to expand...
But how can I scrape it by category, if the category is not in the url? The sitemap is in this format: domain.com/article-title
 
Almostbr0ke said:
But how can I scrape it by category, if the category is not in the url? The sitemap is in this format: domain.com/article-title
Click to expand...
in this case you have to access every url and scrape the category from from the article's page (assuming the category is provided in the article)
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top