My advice to you would be to learn a little Python (my experience learning Python has been surprisingly easy)...and you really only need to learn enough to accomplish your needs. There are so many good resources available freely to learn and "Diving InTo Python" is one of them. It's free to read online or a down-loadable pdf, there's tons of example code that's also down-loadable to follow along with the exercises in the book.
In fact here's a pc. of example "Dive InTo Python" code to grab every link on a given page, it runs at warp speed and in literally a second or two it'll return several hundred links or however many links are on that page:
import urllib, urllister
usock = urllib.urlopen("http://your-URL-to-scrape-goes-here.com")
parser = urllister.URLLister()
parser.feed(usock.read())
usock.close()
parser.close()
for url in parser.urls: print url
Or 4 lines with the lxml module:
from lxml.html import parse
doc = parse('http://whatever-the-URL-is.com').getroot()
for link in doc.cssselect('div.pad a'):
print '%s: %s' % (link.text_content(), link.get('href'))
However these examples don't follow the links, the point is just to show you how easy it is, and only takes a few lines of code to get all the links on a given page. The possibilities are really endless with Python and it's saving me so much time and automating alot of my tasks.
There are Modules that will follow all the links or just ones that you specify and can go unlimited levels deep. That's what it sounds like your looking for, and the Scrapy Module should do that nicely for you. And all on auto pilot, schedule it to run whenever you want, have it write the data into spread sheets or a mysql data base, whatever you want. They have a very active community as well in google groups where everyone helps each other out and you can get support.
Check it out...It's not hard at all to learn a little bit of Python and it's always nice to be able to customize your own tools especially the way things change so fast all the time anymore.
