Search results

  1. S

    Free Web Site Crawling [LARGE!]

    Hi all! I'm developing web crawler for big websites (1-10 millions pages) more than half a year. Now I testing it on everything. What I store: - page info: titles, desc, headers, status code, load time, distance to index, out links, in links, more than 20 attributes - links: anchor, follow...
Back
Top