How to Spider a Website...

turbopugsleylx

Elite Member

Jun 6, 2008

3,424

988

• Feb 6, 2010

• #1

Hey guys I am looking for a website spider or scraper where I give it a domain and it'll spit out a list of all the URLS that are within that domain...how do I do this? Is there a software or site that will accomplish this? Like for instance if I wanted to get a entire list of all the URL's and subpages within the domain www.ask.com how would I got about doing this?

 

gregstereo

Elite Member

Jr. VIP

Oct 5, 2009

2,356

1,201

• Feb 6, 2010

• #2

I like Xinu's Link Sleuth. Free, been around a long time, efficient.

 

nufaman

Elite Member

May 29, 2009

1,870

1,345

• Feb 6, 2010

• #3

Yeah, I use that one too
home.snafu.de/tilman/xenulink.html

 

turbopugsleylx

Elite Member

Jun 6, 2008

3,424

988

• Feb 6, 2010
• Thread Starter Thread Starter

• #4

When I try to scrape a forum why is it only giving me like 20 urls? (and yes im logged in)

 

cooooookies

Senior Member

Oct 6, 2008

1,133

283

• Feb 7, 2010

• #5

Check htdig.org...

When you scrape with a tool a forum which does not show content publicly, you won't get too many results regardless if you are logged in or not via your browser, since the one has nothing to do with the other. And: when you scrape too fast, your IP will be banned. This is nowadays almost matter-of-fact for all huge high ranking sites.

 

mpkfa

Newbie

Feb 3, 2010

23

3

• Feb 8, 2010

• #6

thanks nufaman