[Free Script] Bulk Keyword Deduplication / Similarity Check

[John Willusa]

Great sharing .
Thank you so much mate .

 

[EvoCreatix]

I was in need for something like this!
I have learned a lot from this!

 

[BlogPro]

Working on an enhanced version of this script.

Any specific features y'all would like to request?

 

[stephen021]

Hello @BlogPro, I am truly grateful to you for this fantastic script. I have been using this script on a regular basis, and it has proven to be exceptionally proficient in handling a large volume of keywords. I conducted a test with over 50,000 keywords, and it performed without any issues.

I'm excited to hear about your plans to upgrade the script.

Is it possible to add filters within the script? These filters could include:

1. Excluding keywords that do not contain specific word(s).
2. Retaining keywords that contain specific word(s).
3. Eliminating keywords consisting of three or fewer words.
4. Retaining keywords that consist of four or more words.

Additionally, I would like to make another request regarding a different feature or script:

Could you kindly create a script that can group keywords?

Your expertise in this area would be greatly appreciated.

 

[GenesisOne]

Valuable share op.

 

[Deleted member 1232083]

Hello @BlogPro, I am truly grateful to you for this fantastic script. I have been using this script on a regular basis, and it has proven to be exceptionally proficient in handling a large volume of keywords. I conducted a test with over 50,000 keywords, and it performed without any issues.

I'm excited to hear about your plans to upgrade the script.

Is it possible to add filters within the script? These filters could include:

1. Excluding keywords that do not contain specific word(s).
2. Retaining keywords that contain specific word(s).
3. Eliminating keywords consisting of three or fewer words.
4. Retaining keywords that consist of four or more words.

Additionally, I would like to make another request regarding a different feature or script:

Could you kindly create a script that can group keywords?

Your expertise in this area would be greatly appreciated.

Working on an enhanced version of this script.

Any specific features y'all would like to request?

If op you can do what the member suggested will be perfect

 

[BlogPro]

Hello @BlogPro, I am truly grateful to you for this fantastic script. I have been using this script on a regular basis, and it has proven to be exceptionally proficient in handling a large volume of keywords. I conducted a test with over 50,000 keywords, and it performed without any issues.

I'm excited to hear about your plans to upgrade the script.

Is it possible to add filters within the script? These filters could include:

1. Excluding keywords that do not contain specific word(s).
2. Retaining keywords that contain specific word(s).
3. Eliminating keywords consisting of three or fewer words.
4. Retaining keywords that consist of four or more words.

Additionally, I would like to make another request regarding a different feature or script:

Could you kindly create a script that can group keywords?

Your expertise in this area would be greatly appreciated.

Hey,

Am super glad you're using this regularly and it is proficient in your requirements. That's all I wanted to hear.

Thanks for the ideas.

Excluding safe words, retaining safe words and eliminating based on word counts are all doable.

These will happen before the similarity model kicks in. Not a big ask.

As for grouping, yes am working on something that would use a LLM model to achieve that. I am actually deploying Mistral in a day or two to test. Agreed it may not be a the fastest solution, but if a strong language model can run on your local machine and perform excellent grouping, it should be worth it. No?

Cheers. And thanks for the good words man.

 

[stephen021]

Great to hear you're on board with these ideas! Thanks again for your commitment to improving the script, and I'm eagerly awaiting for the updates. Keep up the excellent work!

 

[Dark Hat 007]

Google Colab Notebook
Colab Link​

Saw a lot of you were having difficulty running this script locally, so created a Google Colab Notebook.​

​

Here's how you can use it ​

​

1. Open the Google Colab of the file here - Google Colab Link​

​

2. Make sure you're logged in to your Gmail and click the Copy to Drive button (just below the file name at the top) - if you're unable to find it, just click File >> Save a Copy in Drive​

​

If you do it right, a new Window will open with your file's copy from your Google Drive​

​

3. Now, in the top menu, goto Runtime and click on Run All​

​

//​

​

That's it - your script is now ready to take your Keyword file and remove similar keywords. ​

Here comes the fun part - ​

​

1. Step 1 of the file will download and install all dependencies.​

2. Step 2 - this is where it will ask you to upload your keywords.csv - just click on "Choose Files" and upload. ​

3. Once that is done, scroll to the very bottom and watch as the script runs its magic, generates the necessary embeddings etc. ​

4. Once it is done - it will ask you 2 questions​

​

1. Enter the similarity threshold (0 to 1, default: 0.8):​

​

Think of it like a percentage. So if you enter 0.90 - you're telling the script "Remove any keyword that is 90% or more similar to the others". If you enter "0.80" - you're asking for 80% or more - so on and so forth.​

​

2. Enter the output Excel file name (default: keywords_output.xlsx):​

​

Whatever you want the output file name to be. You can just hit enter.​

​

That's it you're done!​

​

To access your done files, simply click on the "File" icon (Fourth icon resembling a directory to the very left of your screen)​

​

And you should see keywords_output.xlsx and unique_keywords.csv there. Simply right-click and download. ​

​

Let me know if you have any questions or get stuck.​

Damn!!! I love BHW!

 

[Dark Hat 007]

Google Colab Notebook
Colab Link​

Saw a lot of you were having difficulty running this script locally, so created a Google Colab Notebook.​

​

Here's how you can use it ​

​

1. Open the Google Colab of the file here - Google Colab Link​

​

2. Make sure you're logged in to your Gmail and click the Copy to Drive button (just below the file name at the top) - if you're unable to find it, just click File >> Save a Copy in Drive​

​

If you do it right, a new Window will open with your file's copy from your Google Drive​

​

3. Now, in the top menu, goto Runtime and click on Run All​

​

//​

​

That's it - your script is now ready to take your Keyword file and remove similar keywords. ​

Here comes the fun part - ​

​

1. Step 1 of the file will download and install all dependencies.​

2. Step 2 - this is where it will ask you to upload your keywords.csv - just click on "Choose Files" and upload. ​

3. Once that is done, scroll to the very bottom and watch as the script runs its magic, generates the necessary embeddings etc. ​

4. Once it is done - it will ask you 2 questions​

​

1. Enter the similarity threshold (0 to 1, default: 0.8):​

​

Think of it like a percentage. So if you enter 0.90 - you're telling the script "Remove any keyword that is 90% or more similar to the others". If you enter "0.80" - you're asking for 80% or more - so on and so forth.​

​

2. Enter the output Excel file name (default: keywords_output.xlsx):​

​

Whatever you want the output file name to be. You can just hit enter.​

​

That's it you're done!​

​

To access your done files, simply click on the "File" icon (Fourth icon resembling a directory to the very left of your screen)​

​

And you should see keywords_output.xlsx and unique_keywords.csv there. Simply right-click and download. ​

​

Let me know if you have any questions or get stuck.​

Hello, please why do I get this error?

 

[BlogPro]

Hello, please why do I get this error?

View attachment 288783

Hey - sorry I missed this. Are you still facing this issue? Can you check with a different CSV file?

//

The new script should be ready in a week or so with the upgrades requested. And I am also making some progress with the clustering script.

 

[Dark Hat 007]

The new script should be ready in a week or so with the upgrades requested. And I am also making some progress with the clustering script.

Let me just wait for the new script.

 

[Beklink]

Thanks for sharing this man

 

[BlogPro]

@MonkeyJohn, @netkam, @banto123, @C63, @Moschoutis

You wanted the clustering script.

https://www.blackhatworld.com/seo/f...ustering-script-using-topic-modeling.1542973/

 

[Deleted member 1232083]

@MonkeyJohn, @netkam, @banto123, @C63, @Moschoutis

Hi Guys

https://www.blackhatworld.com/seo/f...ustering-script-using-topic-modeling.1542973/

Thanks for the tag

 

[bottingisfine]

Where does the final CSV appear?

amazing stuff, I just can find the final file :/

 

[BlogPro]

Where does the final CSV appear?

amazing stuff, I just can find the final file :/

Did you run it on Colab or locally?

On colab, look for the file/folder icon to the very left of the screen. You should then see the files.

 

[Elaine12]

Thanks For Sharing

 

[bottingisfine]

Did you run it on Colab or locally?

On colab, look for the file/folder icon to the very left of the screen. You should then see the files.

thanks! great stuff

 

[geomink]

@BlogPro

I get this warning after the similarity threshold and before the output:

C:\Users\******************\AppData\Local\Programs\Python\Python311\Lib\site-packages\sklearn\cluster\_agglomerative.py:1006: FutureWarning: Attribute `affinity` was deprecated in version 1.2 and will be removed in 1.4. Use `metric` instead
warnings.warn(

Is there anything I need to do?
Many thanks