Need all Urls from given site
- Thread starter dolphin1
- Start date
- Oct 4, 2011
- 199
- 1,038
What you exactly mean with "all the urls".
Every single fucking link on that page or just related servers and sub adresses for the main page?
Every single fucking link on that page or just related servers and sub adresses for the main page?
If you have access to a linux command line, 'wget' can do what you asked with it's -r (recursive) option.
It does, however, pull down ALL THE CONTENT from the website in question, which may or may not be a bonus for you.
You'll want to use the -r option for recursive, and the -o option to specify a logfile (where you can easily extract the url strings that were downloaded ). And you'll want to make sure you have enough local disk space...
It does, however, pull down ALL THE CONTENT from the website in question, which may or may not be a bonus for you.
You'll want to use the -r option for recursive, and the -o option to specify a logfile (where you can easily extract the url strings that were downloaded ). And you'll want to make sure you have enough local disk space...
elclandestino
Registered Member
- Feb 29, 2012
- 61
- 7
If you are using Linux or OSX, i can give you a simple tool that I wrote in Python
yep, any method is based on site scraping. you should use (or write yourself) some kind of web spider which will follow site links (filter external) and save it to database or file.
Last edited:
Khalil Cazoo
Newbie
- Apr 10, 2012
- 8
- 2
It's a walk in the park to write a bot for that. You just have to scrape the homepage and store all links. For each stored link, download the content and store then available links again. And so on, until there's not link left. It's a simple loop! 
thom.holland
Newbie
- Apr 23, 2012
- 16
- 0
Use PHP+regex? I can even make it for you if you want, just shoot me a PM with details
hindsight46
Registered Member
- Oct 7, 2011
- 80
- 13
You can use a simple footprint in scrapebox to grab all urls. I used it to grab only certain inner page urls and it worked like a charm. Programming a bot would be useful too.