I am reviving this thread with a new update.
Towards the end of 2021, I had finished the MVP version of my plugin "CrawlSpider SEO monitor". After that I installed on several of my websites and few testers. I saw some success in marketing. Later due to change in priorities and in order to help my wife's business, I jumped into Etsy and Amazon selling personalized art items. I was dedicating most of my time helping her setup the process, setting up the shops, websites etc. Now she is running the show by herself. Now I am getting back into the groove and continue the plugin journey.
When I bought the crawlspider domain, I had the intention to build a very grand SEO Audit tool. That is still my long term goal. I needed the Audit tool to have the "Diff" functionality where it can automatically tell you what changed between audits so I embarked on building that scalable architecture that can run on wordpress.
My plan is
1. CrawlSpider: SEO Change Monitor - Done
2. CrawlSpider: Auto Linker - WIP
3. CrawlSpider: SEO Auditor - Planned
I am now documenting my build and journey for WordPress Internal Auto linker. Using the same architecture as in the "SEO Change Monitor", it is very easy to add modules and that is where my focus is currently.
More about the Auto Linker plugin
1. The reason why I call it the Auto Linker is because that is my primary focus. It will use combination of AI tech to determine the match and implement links based on the rules. But it will also allow you to easily monitor what was implemented and undo them very easily. It will also tell you exactly why it thinks the internal link was implemented.
2. Provide manual inter linking process just like other popular plugins like LinkWhisper but my primary focus is to build the UI that is geared towards automation and totally hands off approach.
3. Collect metrics
4. Provide stunning reports
Scalable Architecture
Let me describe what this means with respect to wordpress and hosting services with limited CPU execution time
1. Majority of the wordpress hosting is on shared servers and even the dedicated ones have a PHP execution timeout.
2. This timeout varies but let say it is set to 30 seconds.
3. So if you were to scan a page, extract the contents, run various checks on them then it is possible that 30 seconds maybe just enough to process few pages.
4. For e.g in the SEO Monitor, it has various checks like https redirect, certificate, sitemap, missing Titles, H1, H2, anchor tags and so on.
5. Each page goes through say 20 individual checks.
6. The plugin database architecture allows for micro processing of these individual checks and has the ability to resume the checks where it last left off.
7 . Say for e.g It was scanning page2 and stopped at check #4 due to a PHP timeout or server crash. The next time it resumes it will pick from the same point and complete the scan.
8. Auto linker will use the same mechanics and this will enable to scan potentially large websites.
Stay tuned for more updates...