Any tools for scrap all URLs from a site?

F

Fardin2012

Regular Member
Joined
Aug 5, 2015
Messages
398
Reaction score
160
Hi, normally when I need to I scrap URL's from a targeted site I scrap from sitemap but I found many sites block there sitemaps for public access. So is there any other way to scrap all URL's from a site?
 
Advertise on BHW
Try these -

1) WP Scraper Pro
2) WP Robot
3) Scraper - Automatic Content Crawler Plugin for WordPress (Envato)

If you find any success with any of them pls let the community know. It will be helpful for others too.

All the best!
 
Fardin2012 said:
Hi, normally when I need to I scrap URL's from a targeted site I scrap from sitemap but I found many sites block there sitemaps for public access. So is there any other way to scrap all URL's from a site?
Click to expand...
Change your user agent to GoogleBot so you can get access to the sitemap. Easier compared to what you are looking for.
 
changing user agent not worked and I dont have Scrapebox, so now I am testing a python script, if this work then I will share here.
 
Fardin2012 said:
Hi, normally when I need to I scrap URL's from a targeted site I scrap from sitemap but I found many sites block there sitemaps for public access. So is there any other way to scrap all URL's from a site?
Click to expand...

a little advice from me, maybe you can try the chrome extension "AnyPicker", if only to do light scrape link task, but for more complex scrapebox is the best choice ;)
 
if you take the urls from the sitemap, you can miss urls if they are not listed there. Sitemap just shows what the webmaster wants.

So the best option for me would be Screamming Frog.
 
Fardin2012 said:
changing user agent not worked and I dont have Scrapebox, so now I am testing a python script, if this work then I will share here.
Click to expand...

If your custom script doesn't do what you need, perhaps try this scraper: https://webscraper.io/

Though, I recommend that you purchase Scrapebox. It has many different features aside from the one you need it for: https://www.blackhatworld.com/seo/s...auto-blog-commenter-with-prstorm-mode.129096/

It's $77 with their BHW discount that's at the bottom of them OP.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top