tool to extract urls from websites needed

Discussion in 'Black Hat SEO Tools' started by Claudiu, May 11, 2011.

  1. Claudiu

    Claudiu Regular Member

    Joined:
    Feb 28, 2009
    Messages:
    312
    Likes Received:
    91
    Location:
    127.0.0.1
    hi,

    looking for a tool that can generate a list of urls from a website.
    pretty much like a sitemap tool.

    the issue with sitemap tools is that it checks if the connection with the pages is good. i don't like this as it takes to much time, which i don't have cause of my big list of websites to crawl.

    so, a tool that can check for urls only (and not the connectivity) is what i'm interested in.

    i'd appreciate your input on this if you know of a tool that can help me.
     
  2. banel

    banel Regular Member

    Joined:
    Mar 30, 2010
    Messages:
    287
    Likes Received:
    16
    You can use a scraper and use site:www.site.net as a footprint.

    P.S.: esti roman nu ?
     
  3. zmoney

    zmoney Senior Member

    Joined:
    Apr 19, 2011
    Messages:
    813
    Likes Received:
    250
    Download "link gopher" plugin for firefox. Simply click "extract all links" and it gives you a list of the urls :)

    EDIT: oh, you want a spider .... this wont do that, just the current page..
     
  4. flexnds

    flexnds Power Member

    Joined:
    Jan 4, 2010
    Messages:
    643
    Likes Received:
    682
    Occupation:
    Internet Marketing, Web development, Internet Repu
    Location:
    AZ
    HTML:
    http://www.xml-sitemaps.com/
    Up to 500 pages for free. You could also use scrape box with site:domain.com
     
  5. sikandar

    sikandar Senior Member

    Joined:
    Mar 15, 2008
    Messages:
    1,160
    Likes Received:
    1,042
    The best tool I know for this is "Web Data Extractor".
     
  6. fb_pva_codes

    fb_pva_codes Newbie

    Joined:
    May 16, 2011
    Messages:
    6
    Likes Received:
    1
    i dont know any soft, but if you have still problem - pm me - i can make for you script/bot

    or type in google 'sitemap creator'