How I can get all pages from URL like this?

C

ChrisLi

Newbie
Joined
Feb 27, 2024
Messages
21
Reaction score
2
Hi, Guys,
Can any body share a strategy that getting all pages from URL like : http://www.xxx.com/show.php?id=5q7uny8u http://www.xxx.com/show.php?id=3t5n8kmg http://www.xxx.com/show.php?id=8i9qfr0n

Can I get them from Semrush or I have to make a tool?
Thank you
 
Advertise on BHW
You can use SEMrush or Screaming Frog to find all the pages on your site, including those with dynamic URLs. If these tools don’t cover everything, you might need a custom script or tool to get all the URLs.
 
You can use the Screaming Frog SEO Spider tool to extract all the page URLs
 
ChrisLi said:
Can I get them from Semrush or I have to make a tool?
Click to expand...
If the URLs that you want to collect are hyperlinked on the given pages, then SEMrush, Screaming Frog or any website audit tool can collect the URLs provided that you have enough level of subscription in which all pages can be crawled.
There are also some plugins that can fetch internal and external links on pages so if there are few pages only then you can use that as well.

You can build your own customized tool, then you can do that in python with libraries like BeautifulSoup

If URLs are in text format then it could be more tricky as you will need to process text and extract URLs with match pattern through regex.
 
For some websites you won't be able to do that. The reason is that the parameter in the URL is a token or something similar.
It could also be a secret link to share a resource or similar. Think sharing links. Or links to confirm your email.
 
If the URLs are linked somewhere on your site, you can use tools like Screaming Frog to crawl and collect them. However, if the URLs are dynamically generated or aren't linked anywhere, you might need to create a custom script to scrape them. For that, using Python with libraries like BeautifulSoup or Scrapy can help you extract these URLs by crawling through the site or matching specific URL patterns.
 
If you need detailed information about pages with specific query parameters, you might need a custom tool. For instance, you could build a web scraper or use existing scraping tools to crawl the site and extract URLs based on the pattern you're interested in.If you don't have a tool, starting with Semrush or a similar SEO tool is a good way to get a broad view, and if you need more specific data, you might consider custom scraping solutions.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top