SuperLinks
Elite Member
- Jul 14, 2008
- 2,903
- 852
I'm looking to see what the best tools or methods there are to scraping the external URLs on a website? I've tried using Xenu Link Sleuth but am running into problems crawling sites on my VPS as I'm running out of memory since Xenu also documents all internal URLs, and information about them thus becoming a memory hog rather than an effective tool for this.
I've also tried Scrapebox by scraping the indexed URLs and then using the external URL tool, however this method only works for indexed domains and indexed URLs. Both of which can be a problem for this task.
Is anyone aware of a tool or method to scrape the external URLs from a website efficiently?
I've also tried Scrapebox by scraping the indexed URLs and then using the external URL tool, however this method only works for indexed domains and indexed URLs. Both of which can be a problem for this task.
Is anyone aware of a tool or method to scrape the external URLs from a website efficiently?