1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How to Spider a Website...

Discussion in 'Black Hat SEO Tools' started by turbopugsleylx, Feb 6, 2010.

  1. turbopugsleylx

    turbopugsleylx Jr. VIP Jr. VIP Premium Member

    Joined:
    Jun 6, 2008
    Messages:
    3,151
    Likes Received:
    965
    Occupation:
    www.xgcmedia.com
    Location:
    www.xgcmedia.com
    Home Page:
    Hey guys I am looking for a website spider or scraper where I give it a domain and it'll spit out a list of all the URLS that are within that domain...how do I do this? Is there a software or site that will accomplish this? Like for instance if I wanted to get a entire list of all the URL's and subpages within the domain www.ask.com how would I got about doing this?
     
  2. gregstereo

    gregstereo Elite Member

    Joined:
    Oct 5, 2009
    Messages:
    1,833
    Likes Received:
    1,027
    Occupation:
    I'm known to locate certain things from time to ti
    Location:
    Moose Factory, ON
    I like Xinu's Link Sleuth. Free, been around a long time, efficient.
     
  3. nufaman

    nufaman Elite Member

    Joined:
    May 29, 2009
    Messages:
    1,697
    Likes Received:
    1,185
    Yeah, I use that one too
    home.snafu.de/tilman/xenulink.html
     
  4. turbopugsleylx

    turbopugsleylx Jr. VIP Jr. VIP Premium Member

    Joined:
    Jun 6, 2008
    Messages:
    3,151
    Likes Received:
    965
    Occupation:
    www.xgcmedia.com
    Location:
    www.xgcmedia.com
    Home Page:
    When I try to scrape a forum why is it only giving me like 20 urls? (and yes im logged in)
     
  5. cooooookies

    cooooookies Senior Member

    Joined:
    Oct 6, 2008
    Messages:
    1,008
    Likes Received:
    216
    Check htdig.org...

    When you scrape with a tool a forum which does not show content publicly, you won't get too many results regardless if you are logged in or not via your browser, since the one has nothing to do with the other. And: when you scrape too fast, your IP will be banned. This is nowadays almost matter-of-fact for all huge high ranking sites.
     
  6. mpkfa

    mpkfa Newbie

    Joined:
    Feb 3, 2010
    Messages:
    23
    Likes Received:
    3
    thanks nufaman