1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How to extract url list from txt file?

Discussion in 'BlackHat Lounge' started by agag2, May 31, 2013.

  1. agag2

    agag2 Supreme Member

    Joined:
    Feb 17, 2009
    Messages:
    1,308
    Likes Received:
    254
    Hi

    I'm wondering if there's a software that will scan a txt file and create a list of urls in txt file..? Or scan a web page and extract all urls on web page and create list of ulrs..?

    Thanks
     
  2. shaggy93

    shaggy93 Jr. VIP Jr. VIP Premium Member

    Joined:
    Dec 23, 2010
    Messages:
    1,027
    Likes Received:
    449
    Location:
    0.0.0.0
    regex can do that for you!
     
    • Thanks Thanks x 1
  3. SmartMan

    SmartMan BANNED BANNED

    Joined:
    Jul 25, 2012
    Messages:
    673
    Likes Received:
    1,244
    This little piece of software does the job :

    Code:
    http://www.focalmedia.net/urlextract.html
    You can use Scrapebox's link extractor add-on to extract links from the Web pages.
     
  4. agag2

    agag2 Supreme Member

    Joined:
    Feb 17, 2009
    Messages:
    1,308
    Likes Received:
    254
    any online tool that works?
     
  5. cooltoad

    cooltoad Senior Member

    Joined:
    Sep 10, 2010
    Messages:
    934
    Likes Received:
    549
    Occupation:
    None of your business
    Location:
    On Vacation
    Make a quick WP post with the text file and then enable SEO quake tool bar. In the top right cornet you will be able to see Links - External and Internal. Click on external and everything will be neatly organized in a CSV file. Quick 2 minutes work.
     
  6. hellnation

    hellnation Regular Member

    Joined:
    Feb 15, 2013
    Messages:
    278
    Likes Received:
    62
    Occupation:
    Senior Network and System Engineer
    Location:
    Quebec, Canada
    use grep with a regular expression...

    powergrep on windows is good.
     
  7.  ﴾͡๏̯͡๏﴿.tk

     ﴾͡๏̯͡๏﴿.tk Power Member

    Joined:
    Oct 6, 2010
    Messages:
    682
    Likes Received:
    114
    Occupation:
    ﴾͡๏̯͡๏﴿ ───█ ﴾͡●
    Location:
    twitter.com/SeX#.﴾͡๏̯͡๏﴿.
    Home Page:
    the posted link focalmedia.net/urlextract.html is death

    with the chrome extension Link klipper it's easy to extract all urls from webpages or the links from a selected area.

    there are also options ?regex and ?multiple extension.

    Q: what is the regex for extracting only the "http://twitter.com/urlS" or better only the /handle ?
     
    Last edited: Oct 31, 2015
  8. RichyA1

    RichyA1 BANNED BANNED

    Joined:
    Oct 31, 2015
    Messages:
    28
    Likes Received:
    7

    This :) Regex is what you need.