Website pages scraper needed. Cant seem to find one.

Discussion in 'Black Hat SEO Tools' started by Mrvegas, Jan 14, 2014.

  1. Mrvegas

    Mrvegas Newbie

    Joined:
    Oct 9, 2009
    Messages:
    30
    Likes Received:
    15
    Occupation:
    Design Director
    Location:
    Europe
    Its actually very simple.
    I'm looking for a piece of software that follows all urls on the website and generates a list of all the pages from the website/domain.
    Just like a sitemap. But then just a simple list. in csv or txt.

    I'm using Integrity right now. which is ok. but I would like to run simultanious domains websites. So I can scan all at once. Just leave it for the night.

    Anyone know a good application? Win, Mac doesn't matter as long as it can scape all urls/pages
     
  2. trajek

    trajek Newbie

    Joined:
    Nov 29, 2010
    Messages:
    45
    Likes Received:
    10
    Gender:
    Male
    Location:
    Austin, Tx U.S.A
    Home Page:
    Have you taken a look at Screaming Frog SEO Spider?
     
  3. macdonjo3

    macdonjo3 Jr. VIP Jr. VIP

    Joined:
    Nov 8, 2009
    Messages:
    5,787
    Likes Received:
    4,469
    Location:
    Toronto
    So you want a list of all links on a URL? Or do you want a list of all links on all links of a URL (depth: 2)?
     
  4. seeplusplus

    seeplusplus Power Member

    Joined:
    Aug 18, 2008
    Messages:
    520
    Likes Received:
    167
    wget is available for Windows.
     
  5. misterblackhat

    misterblackhat Newbie

    Joined:
    Jan 17, 2014
    Messages:
    19
    Likes Received:
    1
    Occupation:
    SEO specialist
    Location:
    US
    Like the user said above, try screaming frog