Hi, new here. Long term active in SEO. A year ago, I started training my own deep learning models and using (chaining) API’s (like GPT3) to make my SEO life a lot easier.
Recently I tested a python library called trafilatura. One of its features is that in can extract the main content of a given url. It works very well so far. (tested on 50+ landing pages, mostly blogs. But also product pages) It filters out headings, footer text etc.
Then I tried this: Many new customers have allot of “almost” duplicate landing pages. But screamingfrog and other seo tools wont filter those out because sentences are rewritten, or changed order.
So I created a python script which:
Even when I shuffle the content 180 degrees this script still gives a 99% content match. So I guess if I can do this, Google must be doing this (no brainer).
I need some AI projects to work on and in this case I can expand this script further to crawl and check complete websites for content similarity.
Yet, it’s allot of work so my first question: are there any 'smart' tools doing this?
Maybe you have other suggestions. If so, I’ll share all the code here for those who are interested. Even if you’re not a programmer, it’s easy to implement.
Thanks!
Recently I tested a python library called trafilatura. One of its features is that in can extract the main content of a given url. It works very well so far. (tested on 50+ landing pages, mostly blogs. But also product pages) It filters out headings, footer text etc.
Then I tried this: Many new customers have allot of “almost” duplicate landing pages. But screamingfrog and other seo tools wont filter those out because sentences are rewritten, or changed order.
So I created a python script which:
- Extracted the main content of 2 similar landing pages with trafilatura
- Used a BERT sentenceTransformer for tokenizing this content (so I get 1 Vector for each landingpage)
- Use cosine similarity to compare these 2 vectors
Even when I shuffle the content 180 degrees this script still gives a 99% content match. So I guess if I can do this, Google must be doing this (no brainer).
I need some AI projects to work on and in this case I can expand this script further to crawl and check complete websites for content similarity.
Yet, it’s allot of work so my first question: are there any 'smart' tools doing this?
Maybe you have other suggestions. If so, I’ll share all the code here for those who are interested. Even if you’re not a programmer, it’s easy to implement.
Thanks!
