Hi all!
I'm developing web crawler for big websites (1-10 millions pages) more than half a year. Now I testing it on everything.
What I store:
- page info: titles, desc, headers, status code, load time, distance to index, out links, in links, more than 20 attributes
- links: anchor, follow...