1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Crawl a site to find all subdomains and pages

Discussion in 'Black Hat SEO Tools' started by mate73, Aug 12, 2010.

  1. mate73

    mate73 Junior Member

    Joined:
    Jul 16, 2010
    Messages:
    110
    Likes Received:
    184
    Occupation:
    Self-employed student
    Location:
    Wherever the trail may lead
    I'm looking for a tool to crawl a MU blog with thousands of subdomains and pages. I started out using Scrapebox but it's simply impossible since there's thousands of new pages published every day.

    I'm basically looking for a tool in which you can crawl a domain (subdomains), crawl a RSS-feed every 30 minutes and if possible, check PR. I'm guessing the last two parts shouldn't be that difficult so I'm asking you guys for help.
     
  2. bbrez1

    bbrez1 Power Member

    Joined:
    Feb 21, 2009
    Messages:
    675
    Likes Received:
    2,360
    This website has some nice crawlers:
    Code:
    http://www.google.com/search?client=firefox-a&rls=org.mozilla%3Asl%3Aofficial&channel=s&hl=en&source=hp&q=website+crawler&btnG=Google+Search
    Seriously tho, I found some for free here. Try them out.

    DRKspider does not look bad