best free crawler for small website?

W

wizeman

Power Member

Aug 15, 2017

757

90

• Jan 31, 2019

• #1

please and thanks

 

Maxim Alston

Junior Member

Oct 25, 2018

129

41

• Jan 31, 2019

• #2

Screaming Frog. Free for crawls up to 500 URLs.

 

S

silentcrowd

Newbie

Feb 1, 2019

5

2

• Feb 1, 2019

• #3

Compared to the frog there are better options out there.

Consider linkchecker. It's opensource, get it from wummel dot github dot io/linkchecker if you need an exe.
Otherwise just apt install linkchecker to use it from the command line.

 

SEO JOKER

Elite Member

Jr. VIP

Aug 16, 2014

3,085

679

• Feb 1, 2019

• #4

Many crawlers are out there. I think Screaming frog is the best crawler for small websites.

 

goodbood

Regular Member

May 7, 2014

369

81

• Feb 1, 2019

• #5

My vote goes to Screaming Frog it works like a charm !

 

Ailsa10

Newbie

Jan 14, 2019

21

4

• Feb 1, 2019

• #6

CocoScan - provides 1000 free URLs to crawl

 

S

signorm68

Registered Member

Aug 13, 2008

65

12

• Feb 4, 2019

• #7

Cheap VPS and httrack is my choice, maybe a bit advanced but nothing too much complicated.

 

Gogol

Administrator

Staff member

Moderator

Executive VIP

Jr. VIP

Sep 10, 2010

11,812

26,653

• Feb 4, 2019

• #8

Python and selenium. With a bit of technical knowledge (youtube has gazillions of tutorials in case you are interested) it's as easy as setting up your expensive, fancy crawler bot and more importantly; it is FREE and OPENSOURCE!!

 

S

signorm68

Registered Member

Aug 13, 2008

65

12

• Feb 4, 2019

• #9

Gogol said:

Python and selenium. With a bit of technical knowledge (youtube has gazillions of tutorials in case you are interested) it's as easy as setting up your expensive, fancy crawler bot and more importantly; it is FREE and OPENSOURCE!!

Click to expand...

Yap, I agree with you, maybe that is only matter of taste Anycase I prefer Httrack because I am using it for years and it has some great "fine tuning" for advanced usage...