1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Is there a link crawler that can keep going and going?

Discussion in 'BlackHat Lounge' started by karma05, Jul 10, 2016.

  1. karma05

    karma05 Regular Member

    Joined:
    May 30, 2013
    Messages:
    383
    Likes Received:
    17
    Location:
    Ramsdale
    I'm trying to find a link crawling tool which can start at an input url and gather up all links found on the page, then go to each of those and gather up all on those pages, and just keep going and going, then maybe when I decide to pause it, it can start off where it left.

    Is there such a thing?
     
  2. main character

    main character Registered Member

    Joined:
    Jun 18, 2016
    Messages:
    73
    Likes Received:
    28
    Yeah, what you described is a Web Spider / Crawler. It's been done and I'm sure there are members here who have one of their own, but I don't know of any public ones, you will most likely have to hire a developer / programmer to build it out for you.
     
    • Thanks Thanks x 1
  3. JustUs

    JustUs Power Member

    Joined:
    May 6, 2012
    Messages:
    626
    Likes Received:
    585
    Last edited: Jul 10, 2016
  4. sumitgupta1992

    sumitgupta1992 Junior Member

    Joined:
    Jul 20, 2013
    Messages:
    176
    Likes Received:
    26
    Gender:
    Male
    Domain Reanimator is exactly what you are looking for.Although it is highly expensive, it would do the exact job you are looking for.
     
  5. laur.laurix

    laur.laurix Power Member

    Joined:
    May 8, 2013
    Messages:
    702
    Likes Received:
    261
    Occupation:
    Reverse Engineering Maniac
    Location:
    Mars
    Xenu , scrapebox, the crawler made by jamie3000 (a member here)