interkul
Senior Member
- May 21, 2017
- 906
- 95
Hi guys, what I'm looking for is web spider, working on linux (Windows is also acceptable), that works on many threads.
What he should do:
analyze a list of different domains, build sitemap for all of them, search for specific regex on every page and save urls that contain that regex.
If there is no any, he can just make sitemaps for list of my domains with many threads and save them to file.
What he should do:
analyze a list of different domains, build sitemap for all of them, search for specific regex on every page and save urls that contain that regex.
If there is no any, he can just make sitemaps for list of my domains with many threads and save them to file.