Hi all! I'm developing web crawler for big websites (1-10 millions pages) more than half a year. Now I testing it on everything. What I store: - page info: titles, desc, headers, status code, load time, distance to index, out links, in links, more than 20 attributes - links: anchor, follow, index, alt text, image, robots blocked and more I offer a FREE crawl of any website (size less than 10 mln pages). As result I give you CSV, Excel file or access to MySQL database with all contents. Your data will NOT shared to anyone. Just write me PM.