1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Need all Urls from given site

Discussion in 'Other Languages' started by dolphin1, Oct 5, 2011.

  1. dolphin1

    dolphin1 Newbie

    Joined:
    Sep 27, 2011
    Messages:
    8
    Likes Received:
    1
    Is there any method other than google search to get all urls from website
    not just the home page urls but all that resides the site.
    Thanks
     
  2. hurn

    hurn Power Member

    Joined:
    Jan 21, 2009
    Messages:
    692
    Likes Received:
    191
    a1 website analyzer will give you all the links
     
  3. Black.Star

    Black.Star Junior Member

    Joined:
    Oct 4, 2011
    Messages:
    185
    Likes Received:
    1,028
    Occupation:
    IT security specialist
    Location:
    Europe
    What you exactly mean with "all the urls".
    Every single fucking link on that page or just related servers and sub adresses for the main page?
     
  4. linksable

    linksable Newbie

    Joined:
    Oct 27, 2010
    Messages:
    43
    Likes Received:
    20
    Occupation:
    SEO Business Owner
    Location:
    Texas
    Home Page:
    i use Screaming Frog SEO Spider
     
  5. Boris70

    Boris70 Newbie

    Joined:
    Feb 8, 2010
    Messages:
    3
    Likes Received:
    0
    If you need app like that please send me PM, i can make one for you ;)
     
  6. rish3

    rish3 Newbie

    Joined:
    Feb 24, 2012
    Messages:
    8
    Likes Received:
    6
    If you have access to a linux command line, 'wget' can do what you asked with it's -r (recursive) option.

    It does, however, pull down ALL THE CONTENT from the website in question, which may or may not be a bonus for you.

    You'll want to use the -r option for recursive, and the -o option to specify a logfile (where you can easily extract the url strings that were downloaded ). And you'll want to make sure you have enough local disk space...
     
  7. crashus

    crashus Junior Member

    Joined:
    Feb 26, 2012
    Messages:
    196
    Likes Received:
    98
    I used TeleportPRO back in a days to grab all the site and then used 'grep' unix utility to find all <a></a> tags
     
  8. elclandestino

    elclandestino Registered Member

    Joined:
    Feb 29, 2012
    Messages:
    63
    Likes Received:
    6
    Location:
    South of Equator
    If you are using Linux or OSX, i can give you a simple tool that I wrote in Python
     
  9. kouxx

    kouxx Junior Member

    Joined:
    Aug 2, 2008
    Messages:
    160
    Likes Received:
    38
    i thought you where asking to strip the links of a site and i was just writing a script for you to do this. Butyou are looking to scrape the links of a whole website. I can write that for you in php but it would be 50$ as its quite a bit of work. ( and hrs)
     
  10. webninja

    webninja Newbie

    Joined:
    Dec 22, 2008
    Messages:
    2
    Likes Received:
    0
    yep, any method is based on site scraping. you should use (or write yourself) some kind of web spider which will follow site links (filter external) and save it to database or file.
     
    Last edited: Mar 2, 2012
  11. Khalil Cazoo

    Khalil Cazoo Newbie

    Joined:
    Apr 10, 2012
    Messages:
    8
    Likes Received:
    2
    It's a walk in the park to write a bot for that. You just have to scrape the homepage and store all links. For each stored link, download the content and store then available links again. And so on, until there's not link left. It's a simple loop! :)
     
  12. thom.holland

    thom.holland Newbie

    Joined:
    Apr 23, 2012
    Messages:
    16
    Likes Received:
    0
    Use PHP+regex? I can even make it for you if you want, just shoot me a PM with details
     
  13. hindsight46

    hindsight46 Registered Member

    Joined:
    Oct 7, 2011
    Messages:
    73
    Likes Received:
    11
    Location:
    Cali
    You can use a simple footprint in scrapebox to grab all urls. I used it to grab only certain inner page urls and it worked like a charm. Programming a bot would be useful too.