1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

tool to extract urls from websites needed

Discussion in 'Black Hat SEO Tools' started by Claudiu, May 11, 2011.

  1. Claudiu

    Claudiu Regular Member

    Joined:
    Feb 28, 2009
    Messages:
    287
    Likes Received:
    81
    Location:
    127.0.0.1
    hi,

    looking for a tool that can generate a list of urls from a website.
    pretty much like a sitemap tool.

    the issue with sitemap tools is that it checks if the connection with the pages is good. i don't like this as it takes to much time, which i don't have cause of my big list of websites to crawl.

    so, a tool that can check for urls only (and not the connectivity) is what i'm interested in.

    i'd appreciate your input on this if you know of a tool that can help me.
     
  2. banel

    banel Regular Member

    Joined:
    Mar 30, 2010
    Messages:
    287
    Likes Received:
    16
    You can use a scraper and use site:www.site.net as a footprint.

    P.S.: esti roman nu ?
     
  3. zmoney

    zmoney Senior Member

    Joined:
    Apr 19, 2011
    Messages:
    814
    Likes Received:
    250
    Download "link gopher" plugin for firefox. Simply click "extract all links" and it gives you a list of the urls :)

    EDIT: oh, you want a spider .... this wont do that, just the current page..
     
  4. flexnds

    flexnds Power Member

    Joined:
    Jan 4, 2010
    Messages:
    643
    Likes Received:
    680
    Occupation:
    Internet Marketing, Web development, Internet Repu
    Location:
    AZ
    HTML:
    http://www.xml-sitemaps.com/
    Up to 500 pages for free. You could also use scrape box with site:domain.com
     
  5. sikandar

    sikandar Senior Member

    Joined:
    Mar 15, 2008
    Messages:
    1,097
    Likes Received:
    1,004
    The best tool I know for this is "Web Data Extractor".
     
  6. fb_pva_codes

    fb_pva_codes Newbie

    Joined:
    May 16, 2011
    Messages:
    6
    Likes Received:
    1
    i dont know any soft, but if you have still problem - pm me - i can make for you script/bot

    or type in google 'sitemap creator'