Scraping all articles from website?
- Thread starter AuraMarketing
- Start date
Djinny
Newbie
- Jan 11, 2018
- 8
- 5
Hi! Don't know whether it is the right place, but still want to ask for a solution:
Is there any available, out of the box solution to get updates from a list of 1500 websites? I want to receive an alert when some website in this list changes and what has been changed from the last time the software checked these websites. There are many solutions for tracking webpage changes, but it doesn't help me because I don't have a full map of pages available on these websites in the first place. Tried to extract webpages of the sites by scrapping sitemaps, but I quickly found out that sitemaps weren't full in most sites in the list.
Any advice on this one?
Is there any available, out of the box solution to get updates from a list of 1500 websites? I want to receive an alert when some website in this list changes and what has been changed from the last time the software checked these websites. There are many solutions for tracking webpage changes, but it doesn't help me because I don't have a full map of pages available on these websites in the first place. Tried to extract webpages of the sites by scrapping sitemaps, but I quickly found out that sitemaps weren't full in most sites in the list.
Any advice on this one?
terrycody
Elite Member
- Sep 29, 2012
- 6,421
- 3,329
Are you kidding? With that large number of sites and each sites may contains thousands of URLs, I don't know any services can do that.
For URL part, you can use scream frog to get the full URL list of any website, then do whatever you want.