bluehatface
Regular Member
- Oct 19, 2013
- 290
- 125
It wouldn't be too hard if you've a little bit of programming experience - It can be done in any language. Bash script, python or PHP will probably be simplest. Here's the general gist of it... You can probably piece something together pretty quickly from snippets from Stack Overflow.
Call the script periodically via a cron job, or add a loop to the script with a delay for each iteration.
Code:
First Launch: Scrape home page
Extract all internal links and save to array
Filter out any previously scraped from saved file
Check home page for keyword
Loop:
Scrape each URL from above
Search for keyword, and save URL to array if found
Repeat until all URLS scraped
Save all URLs to file for next run
Finished: Do something with the found URLs. Email/save to disk etc.
sagarblack
Junior Member
- Jan 10, 2016
- 127
- 16
use imacros the easyest way
maxsch0610
Newbie
- Jul 27, 2018
- 19
- 4
would be easy, i would say a dev only need like half-full hour
JohnRackham
Newbie
- Aug 24, 2018
- 4
- 1
Not hard at all. What language did you want to program it in?
Regardless, what you'd want to do is create a crawler that looks for your string while crawling the website. If it finds said string, have it insert the url into a database for later retrieval. MySQL if you want to have it online or SQLite if you want to keep the database as a local file.
If it was me, I'd probably use python (possibly with selenium if the site was javascript heavy) and SQLite.
Regardless, what you'd want to do is create a crawler that looks for your string while crawling the website. If it finds said string, have it insert the url into a database for later retrieval. MySQL if you want to have it online or SQLite if you want to keep the database as a local file.
If it was me, I'd probably use python (possibly with selenium if the site was javascript heavy) and SQLite.