Site scraper?

turbopugsleylx

Elite Member

Jun 6, 2008

3,424

988

• Jan 11, 2010

• #1

Hey guys I am looking for a tool or website that will scrape a site and input all the pages that site contains...any ideas? I have looked around but found nothing...

-Turbo

 

CyberWizard

Junior Member

Oct 21, 2009

172

135

• Jan 11, 2010

• #2

Try httrack. it is a full website copier.

 

K

kerxupt

Newbie

Nov 4, 2009

17

0

• Jan 11, 2010

• #3

If you're running Windows, just like CyberWizard mentioned:
+ http://www.httrack.com/

If you're running MacOSX (even iPhone version), try:
+ http://www.sitesucker.us/home.html

If you're running Linux or BSD, try:
+ http://www.gnu.org/software/wget/

If you're actually trying to extract specific parts of a certain website, try using iMacros (http://www.iopus.com/iMacros/)

 

turbopugsleylx

Elite Member

Jun 6, 2008

3,424

988

• Jan 11, 2010
• Thread Starter Thread Starter

• #4

Some of these are good...but I am looking for something basic...something that just scrapes the site and give u a list of links/pages within it....anything else?

 

catman08

Junior Member

Jan 11, 2008

170

110

• Jan 11, 2010

• #5

If you know some php - i could give you my classes i programmed once. with those classes it is a breeze to create any scraper and data analyzer you need.

Let me know via PM if you are interested.
Cheers
catman

 

CoyoteAssassin

Elite Member

Jan 3, 2010

1,868

3,988

• Jan 11, 2010

• #6

I'm currently in the process of doing the same thing with Website Ripper Copier. You can export a list of files. Are you just wanting the URL's? If so, just go online and create a sitemap and use that as your site.

Try this: http://www.microsystools.com/products/sitemap-generator/faq/unlimited-sitemap-generator-pages/

 

T

turokman555

Newbie

Jan 8, 2010

43

1

• Jan 11, 2010

• #7

I can do some custom scraping for you using python for a fee.. PM me...

http://www.url-directory.org

 

turbopugsleylx

Elite Member

Jun 6, 2008

3,424

988

• Jan 12, 2010
• Thread Starter Thread Starter

• #8

You guys know any online only sites that do this?

 

haridada

Senior Member

Oct 9, 2008

1,000

385

• Jun 2, 2010

• #9

CoyoteAssassin said:

I'm currently in the process of doing the same thing with Website Ripper Copier. You can export a list of files. Are you just wanting the URL's? If so, just go online and create a sitemap and use that as your site.

Try this: http://www.microsystools.com/products/sitemap-generator/faq/unlimited-sitemap-generator-pages/

Click to expand...

silly doubt here. Once the html sitemap is generated, is it possible to convert it into text?