[Free Script] Bulk Keyword Deduplication / Similarity Check

[XThomas]

Amazing idea and implementation!
Many thanks!!

 

[BlogPro]

Hate to bother you, but package installation just don't work. I tried on 2 pcs that never had python before, installed latest python both time package installation failed with
ms visual c++ 14.0 missing error
Which is discontinued by MS. Installed the next version, still got the same error.
Can you do a fresh install on some rdp and post the instructions?

Allow me some time to hop on a PC. I'll perform a clean install and let you know work arounds if I get stuck.

 

[seowarrior008]

Hi, just tried to install it in my VPS, but looks like I encountered some errors during the execution,
can OP @BlogPro take a check for me? where I did wrong? thanks a lot in advance

 

[GosuSEO]

Can I share a multithreaded version here?

 

[BlogPro]

Hate to bother you, but package installation just don't work. I tried on 2 pcs that never had python before, installed latest python both time package installation failed with
ms visual c++ 14.0 missing error
Which is discontinued by MS. Installed the next version, still got the same error.
Can you do a fresh install on some rdp and post the instructions?

Hey - I tried installation on a fresh Windows Machine for you. I noticed some issues because the old requirements.txt had version numbers for certain packages.

What I have done is removed the version numbers of packages and post that the installation went by without a hitch.

I tried running the script and everything worked.




You may access the new requirements.txt here - https://github.com/iamBlogPro/Similarity-Check-Advanced/blob/main/requirements.txt

Here is it below -

numpy
pandas
openpyxl
scikit-learn
sentence-transformers

Please run this and let me know if it works now. Or we can try troubleshooting this further.

//

Hi, just tried to install it in my VPS, but looks like I encountered some errors during the execution,
can OP @BlogPro take a check for me? where I did wrong? thanks a lot in advanceView attachment 254539

Hey @seowarrior008 -

This looks like the script is having trouble decoding your keyword.csv file.

Please check for non-unicode character in your keyword list.

This is a common issue when using bulk keyword generators. I normally just check for non-ascii using this tool - https://pages.cs.wisc.edu/~markm/ascii.html

It will output the unrelated encoded character, simply remove them from your list and try again.

 

[IM Dude]

Hey - I tried installation on a fresh Windows Machine for you. I noticed some issues because the old requirements.txt had version numbers for certain packages.

What I have done is removed the version numbers of packages and post that the installation went by without a hitch.

I tried running the script and everything worked.


View attachment 256176

You may access the new requirements.txt here - https://github.com/iamBlogPro/Similarity-Check-Advanced/blob/main/requirements.txt

Here is it below -

numpy
pandas
openpyxl
scikit-learn
sentence-transformers

Please run this and let me know if it works now. Or we can try troubleshooting this further.

//



Hey @seowarrior008 -

This looks like the script is having trouble decoding your keyword.csv file.

Please check for non-unicode character in your keyword list.

This is a common issue when using bulk keyword generators. I normally just check for non-ascii using this tool - https://pages.cs.wisc.edu/~markm/ascii.html

It will output the unrelated encoded character, simply remove them from your list and try again.

Great job! I will redo the install this week.

 

[fay]

what are the maximum number of keywords?
i tried 50k keywords on colab pro, but it crashed everytime

 

[BlogPro]

what are the maximum number of keywords?
i tried 50k keywords on colab pro, but it crashed everytime

The maximum keywords depend on the compute prowess and the available memory in your Colab Pro Compute Engine. The script in itself has no hard stop limit.

I have tried with 22k keywords on my local machine and got by with no issues.

I'd recommend you break down your list into two or three. And then combine them once to run a final time.

 

[seowarrior008]

Hey @seowarrior008 -

This looks like the script is having trouble decoding your keyword.csv file.

Please check for non-unicode character in your keyword list.

This is a common issue when using bulk keyword generators. I normally just check for non-ascii using this tool - https://pages.cs.wisc.edu/~markm/ascii.html

It will output the unrelated encoded character, simply remove them from your list and try again.

Hi bro, thanks for helping out. I just loaded in 10K kws to test things and works perfectly.
Small tool, but can save lots of time while grouping kws. Thanks a lot!

 

[MonkeyJohn]

Thanks again @BlogPro for this script. It is now part of my AI publishing workflow - I use it to reduce the amount of keywords that I import into LowFruits, to save on credits.

Cheers!

 

[BlogPro]

Thanks again @BlogPro for this script. It is now part of my AI publishing workflow - I use it to reduce the amount of keywords that I import into LowFruits, to save on credits.

Cheers!

You have no idea how happy that makes me feel.

So glad that the script is helping you cut costs.

I'll get moving on the clustering script and will release it shortly.

 

[joecal]

@BlogPro Thank you for the amazing work! Definitely will try it out on my current project.
I have been entertaining the idea of creating a similar clustering script but based on Google SERP: I.e. take the top 10 for each of the keywords, and see if they trigger the same set of results. Since serp scraper can add too much to the bill, I am thinking of combining your script with low similarity threshold together with a serp scraper, to see if Google treats these keywords the same. It would give much more precise clustering than distillroberta or Jaccard index. Though, if we enhance Jaccard similarity coefficient with analyzing the similarity based on the neighboring keywords with high semantic relevance, it could bring even better results.

 

[blinks]

thanks, this is really a great idea.

 

[Benji1]

Thanks again @BlogPro for this script. It is now part of my AI publishing workflow - I use it to reduce the amount of keywords that I import into LowFruits, to save on credits.

Cheers!

Why you import than Afterwards into lowfruits?

 

[Benji1]

Im trying to use the script in google collab im getting the error: "IndexError: list index out of range"

 

[Benji1]

Now getting this error: "NameError: name 'csv' is not defined"?

 

[vijay299]

i am working on web tool soon share the url here

 

[seowarrior008]

Hi, just curious of why this script in my VPS can't handle more than 10K keywords. It always ends up with the error once I loaded more than 10k keywords in the CSV sheet.

Any way to fix this? @BlogPro, THANKS a lot in advance!!

 

[vijay299]

finally i created tool but little bit different from it. The sentences transformer taking huge resources and taking time so i created using different method. This also find similarity each other keywords.
I seen lots of online tools but they all works on different algorithm. May be this can work for you. You can request any feature to make more simple and accurate. i cant share the exact url because people will thought i am taking backlink so please remove bracket from url https://wpseohosts(.)com/remove-duplicate-keywords-tool/

 

[vijay299]

please share experience here