Crawl a site to find all subdomains and pages

mate73 · Aug 12, 2010

I'm looking for a tool to crawl a MU blog with thousands of subdomains and pages. I started out using Scrapebox but it's simply impossible since there's thousands of new pages published every day.

I'm basically looking for a tool in which you can crawl a domain (subdomains), crawl a RSS-feed every 30 minutes and if possible, check PR. I'm guessing the last two parts shouldn't be that difficult so I'm asking you guys for help.

bbrez1 · Aug 12, 2010

mate73 said:
I'm looking for a tool to crawl a MU blog with thousands of subdomains and pages. I started out using Scrapebox but it's simply impossible since there's thousands of new pages published every day.

I'm basically looking for a tool in which you can crawl a domain (subdomains), crawl a RSS-feed every 30 minutes and if possible, check PR. I'm guessing the last two parts shouldn't be that difficult so I'm asking you guys for help.

This website has some nice crawlers:

Code:

http://www.google.com/search?client=firefox-a&rls=org.mozilla%3Asl%3Aofficial&channel=s&hl=en&source=hp&q=website+crawler&btnG=Google+Search

Seriously tho, I found some for free here. Try them out.

DRKspider does not look bad

Crawl a site to find all subdomains and pages

mate73

Junior Member

bbrez1

Power Member

Main Menu

Marketplace

Making Money

BlackHatWorld

Other