1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Website pages scraper needed. Cant seem to find one.

Discussion in 'Black Hat SEO Tools' started by Mrvegas, Jan 14, 2014.

  1. Mrvegas

    Mrvegas Newbie

    Joined:
    Oct 9, 2009
    Messages:
    30
    Likes Received:
    15
    Occupation:
    Design Director
    Location:
    Europe
    Its actually very simple.
    I'm looking for a piece of software that follows all urls on the website and generates a list of all the pages from the website/domain.
    Just like a sitemap. But then just a simple list. in csv or txt.

    I'm using Integrity right now. which is ok. but I would like to run simultanious domains websites. So I can scan all at once. Just leave it for the night.

    Anyone know a good application? Win, Mac doesn't matter as long as it can scape all urls/pages
     
  2. trajek

    trajek Newbie

    Joined:
    Nov 29, 2010
    Messages:
    44
    Likes Received:
    10
    Location:
    Austin, Tx U.S.A
    Home Page:
    Have you taken a look at Screaming Frog SEO Spider?
     
  3. macdonjo3

    macdonjo3 Jr. VIP Jr. VIP Premium Member

    Joined:
    Nov 8, 2009
    Messages:
    5,564
    Likes Received:
    4,317
    Location:
    Toronto
    Home Page:
    So you want a list of all links on a URL? Or do you want a list of all links on all links of a URL (depth: 2)?
     
  4. seeplusplus

    seeplusplus Power Member

    Joined:
    Aug 18, 2008
    Messages:
    511
    Likes Received:
    163
    wget is available for Windows.
     
  5. misterblackhat

    misterblackhat Newbie

    Joined:
    Jan 17, 2014
    Messages:
    19
    Likes Received:
    1
    Occupation:
    SEO specialist
    Location:
    US
    Like the user said above, try screaming frog