Server-Based Website-Crawler

D

dddd

Junior Member

Jun 4, 2007

164

161

• May 5, 2009

• #1

Hey guys, does somebody of you know of a good server-based website-crawler?

What I mean by that is:

(a) input URL of website

(b) crawler starts to crawl every single page on the website and saves the page as file

result: a directory on the server with every single page the website has for offline "processing / viewing"

Getting single URL's is pretty easy, but crawling indefinite levels deep is a different thing... :-( So anybody any suggestions please?

 

J

JinxY

Regular Member

May 14, 2007

364

104

• May 5, 2009

• #2

try with wget or curl

 

B

bipingaur

Newbie

Apr 22, 2009

11

0

• May 5, 2009

• #3

Try whttrack it is free and fast

 

I

iintense

BANNED

May 1, 2009

49

58

• May 5, 2009

• #4

Yes, HTTrack is a great program and very fast. It also allows options such as which images to pull and howin depth you want to copy the site.

 

Z

zone69

Junior Member

Nov 24, 2008

196

1,299

• May 5, 2009

• #5

Create a directory to store the site on your server and run wget below, replacing http://somesite.com with the site you wish to download.

Code:

 wget -r -k http:/somesite.com

This will recursively download the site locally.


Or you can download wget for windows and do the same on your windows machine.

Code:

http://gnuwin32.sourceforge.net/packages/wget.htm

Hope that helps.

 

the_demon

Elite Member

Nov 23, 2008

3,220

1,604

• May 5, 2009

• #6

Why server side???

Just Google search backstreet browser. It's a great website mirroring program.

 

D

dddd

Junior Member

Jun 4, 2007

164

161

• May 6, 2009
• Thread Starter Thread Starter

• #7

(a) server-side because it's faster than with my dsl connection
(b) thanks zone69 for the wget idea, didn't know it has crawling options, will try this, again thanks!