1. This website uses cookies to improve service and provide a tailored user experience. By using this site, you agree to this use. See our Cookie Policy.
    Dismiss Notice

scrape a domains URLs

Discussion in 'BlackHat Lounge' started by MarketerMac, Nov 30, 2009.

  1. MarketerMac

    MarketerMac Regular Member

    Oct 26, 2009
    Likes Received:
    I'm searching for (albeit unsuccessfully) a scraper that I can give a domain name to and it will scrape all of the links on domain.com, then follow those links and scrape all the new links it finds until it cannot find anymore links - so in effect I would have every linked to URL from a website. I'm sure one exists, just my searching has only turned up garbage.

    Oh, and if it had proxy support that would be a huge plus! thx provided to all helpful posts!
  2. onlinewealth

    onlinewealth Junior Member

    Mar 13, 2007
    Likes Received:
    Direct marketing working at home.
    "In a State of Corruption"
    I'm not "totally" sure but I believe OptiLink does what you want. I know it shows incoming and outgoing links but you'll have to check it out. They have a demo download.
    • Thanks Thanks x 1