1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

scrape a domains URLs

Discussion in 'BlackHat Lounge' started by MarketerMac, Nov 30, 2009.

  1. MarketerMac

    MarketerMac Regular Member

    Joined:
    Oct 26, 2009
    Messages:
    247
    Likes Received:
    101
    I'm searching for (albeit unsuccessfully) a scraper that I can give a domain name to and it will scrape all of the links on domain.com, then follow those links and scrape all the new links it finds until it cannot find anymore links - so in effect I would have every linked to URL from a website. I'm sure one exists, just my searching has only turned up garbage.

    Oh, and if it had proxy support that would be a huge plus! thx provided to all helpful posts!
     
  2. onlinewealth

    onlinewealth Junior Member

    Joined:
    Mar 13, 2007
    Messages:
    157
    Likes Received:
    101
    Occupation:
    Direct marketing working at home.
    Location:
    "In a State of Corruption"
    I'm not "totally" sure but I believe OptiLink does what you want. I know it shows incoming and outgoing links but you'll have to check it out. They have a demo download.
     
    • Thanks Thanks x 1