Any tools for scrap all URLs from a site?

[Fardin2012]

Hi, normally when I need to I scrap URL's from a targeted site I scrap from sitemap but I found many sites block there sitemaps for public access. So is there any other way to scrap all URL's from a site?

 

[iklan2]

Use scrapebox

 

[sageshark]

Try these -

1) WP Scraper Pro
2) WP Robot
3) Scraper - Automatic Content Crawler Plugin for WordPress (Envato)

If you find any success with any of them pls let the community know. It will be helpful for others too.

All the best!

 

[Alexion]

Hi, normally when I need to I scrap URL's from a targeted site I scrap from sitemap but I found many sites block there sitemaps for public access. So is there any other way to scrap all URL's from a site?

Change your user agent to GoogleBot so you can get access to the sitemap. Easier compared to what you are looking for.

 

[mrbossy]

Scrap box or build your own tool mate !

 

[VNSnowman]

I think Screaming Frog can do the job

 

[JasonS]

Screaming Frog or Scrapebox, both can do the job.

 

[swathi sinha]

Change your user agent to GoogleBot so you can get access to the sitemap. Easier compared to what you are looking for.

Thank you for this

 

[Fardin2012]

Change your user agent to GoogleBot so you can get access to the sitemap. Easier compared to what you are looking for.

I will test this.

 

[larrainc]

Try this :

https://hackertarget.com/extract-links/

 

[She Hulk]

Scrapebox is great one.

 

[Fardin2012]

changing user agent not worked and I dont have Scrapebox, so now I am testing a python script, if this work then I will share here.

 

[Hawkeye Digital]

Scarpbox , you would have to buy it mate !

 

[Riverside]

Hi, normally when I need to I scrap URL's from a targeted site I scrap from sitemap but I found many sites block there sitemaps for public access. So is there any other way to scrap all URL's from a site?

a little advice from me, maybe you can try the chrome extension "AnyPicker", if only to do light scrape link task, but for more complex scrapebox is the best choice

 

[seoñador]

if you take the urls from the sitemap, you can miss urls if they are not listed there. Sitemap just shows what the webmaster wants.

So the best option for me would be Screamming Frog.

 

[PHPInjected]

changing user agent not worked and I dont have Scrapebox, so now I am testing a python script, if this work then I will share here.

If your custom script doesn't do what you need, perhaps try this scraper: https://webscraper.io/

Though, I recommend that you purchase Scrapebox. It has many different features aside from the one you need it for: https://www.blackhatworld.com/seo/s...auto-blog-commenter-with-prstorm-mode.129096/

It's $77 with their BHW discount that's at the bottom of them OP.