[Free Script] Bulk Keyword Deduplication / Similarity Check

[BlogPro]

@BlogPro a small correction for the above. I used your script in google colabs and it works great.. thank you so much mate...

I am glad you got it working my friend

 

[12popi123]

great tool exactly what i needed, thanks

 

[netkam]

I am interested in keyword clustering script with Jaccard Index.

 

[BlogPro]

I am interested in keyword clustering script with Jaccard Index.

Noted.

I am currently in the testing phase for it. I will release it soon.

 

[MKorkut]

@BlogPro a small correction for the above. I used your script in google colabs and it works great.. thank you so much mate...

How did you do this, can you explain to us in detail? I couldn't run it on PC, I guess it's something I don't understand. Or would you help me if I specify KW?

 

[BlogPro]

Google Colab Notebook
Colab Link​

Saw a lot of you were having difficulty running this script locally, so created a Google Colab Notebook.​

​

Here's how you can use it ​

​

1. Open the Google Colab of the file here - Google Colab Link​

​

2. Make sure you're logged in to your Gmail and click the Copy to Drive button (just below the file name at the top) - if you're unable to find it, just click File >> Save a Copy in Drive​

​

If you do it right, a new Window will open with your file's copy from your Google Drive​

​

3. Now, in the top menu, goto Runtime and click on Run All​

​

//​

​

That's it - your script is now ready to take your Keyword file and remove similar keywords. ​

Here comes the fun part - ​

​

1. Step 1 of the file will download and install all dependencies.​

2. Step 2 - this is where it will ask you to upload your keywords.csv - just click on "Choose Files" and upload. ​

3. Once that is done, scroll to the very bottom and watch as the script runs its magic, generates the necessary embeddings etc. ​

4. Once it is done - it will ask you 2 questions​

​

1. Enter the similarity threshold (0 to 1, default: 0.8):​

​

Think of it like a percentage. So if you enter 0.90 - you're telling the script "Remove any keyword that is 90% or more similar to the others". If you enter "0.80" - you're asking for 80% or more - so on and so forth.​

​

2. Enter the output Excel file name (default: keywords_output.xlsx):​

​

Whatever you want the output file name to be. You can just hit enter.​

​

That's it you're done!​

​

To access your done files, simply click on the "File" icon (Fourth icon resembling a directory to the very left of your screen)​

​

And you should see keywords_output.xlsx and unique_keywords.csv there. Simply right-click and download. ​

​

Let me know if you have any questions or get stuck.​

 

[kaluwa1]

How did you do this, can you explain to us in detail? I couldn't run it on PC, I guess it's something I don't understand. Or would you help me if I specify KW?

Hey mate... i was too late to see your request, @BlogPro took my chance .. lol.
Thanks for the well,detailed reply @BlogPro appreciate it alot.

 

[MrAllright]

Cool Script Is it still working?

 

[BlogPro]

Cool Script Is it still working?

I posted it barely a week ago. Of course, it's working.

I also prepared a Google Colab Notebook here to make things easier for you to use - https://www.blackhatworld.com/seo/f...cation-similarity-check.1491191/post-16327593

Have fun. Let me know if you get stuck.

 

[Ameni]

For those having issues with the "huggingface_hub.snapshot_download" dependency, try updating sentence-transformers to the latest version.
That should fix any issues.

pip3 install -U sentence-transformers

 

[celiasieber123]

Can anybody help me!! I'm on ubuntu 22.10.
I'm unable to install packages from requirements.txt.

"pip install -r requirements.txt" gives this message -

"Command 'pip' not found, but can be installed with:
sudo apt install python3-pip"

Thanks.

 

[BlogPro]

Can anybody help me!! I'm on ubuntu 22.10.
I'm unable to install packages from requirements.txt.

"pip install -r requirements.txt" gives this message -

"Command 'pip' not found, but can be installed with:
sudo apt install python3-pip"

Thanks.

Have you installed python?

 

[celiasieber123]

Have you installed python?

Yes Sir,

The "python3 --version" gives me this result "Python 3.10.7"

 

[MoonTrade]

okay let me test this code maybe ill need for future

 

[BlogPro]

Yes Sir,

The "python3 --version" gives me this result "Python 3.10.7"

First check if pip3 is installed -

pip3 --version

If not, run

sudo apt install python3-pip

Check for successful install again with the version command.

Then run

pip3 install -r requirements.txt

 

[celiasieber123]

First check if pip3 is installed -

pip3 --version

If not, run

sudo apt install python3-pip

Check for successful install again with the version command.

Then run

pip3 install -r requirements.txt

Oh yes i realized that and installed pip. The dependencies are installing now. I will check the script soon. Thanks for such swift replies.

 

[celiasieber123]

@BlogPro
Sorry sir one last question. is this normal script behaviour? Its downloading something after executing the script.

 

[BlogPro]

@BlogPro
Sorry sir one last question. is this normal script behaviour? Its downloading something after executing the script.

View attachment 252579

Yes, absolutely normal. It downloads the model the first time you run it.

 

[blackh4t]

Good share thx! Will sort my list of 400 k keyword and see how much left )

 

[celiasieber123]

Yes, absolutely normal. It downloads the model the first time you run it.

Cool.

The script works flawlessly.

Moreover, if you enter 1 as similarity threshold it will weed out exact duplicates.

So many thanks @BlogPro , your script will save 'us' so many tideous hours of mindless manual work.

Also respect+ for guiding a noob.


P.S. Shoutout to @Ameni for pointing out the fix for "huggingface_hub.snapshot_download" dependency issue.