Remove Duplicates 2.5M+ LINES MORE! (HELP)! NOTEBOOK ++

Intofantasy · Oct 30, 2023

Is there a software out there, that can handle more than 2.5 million duplicates?
I'm using notebook ++, I'm trying to remove duplicate lines, in some cases I'm going over 5 or 7 million lines.

I just want to Simply remove the duplicate lines so the list becomes more efficient and smaller.

When I'm using notebook ++, I'm getting error messages " bad allocation" and everything just closes and errors.

daverawcus · Oct 30, 2023

Google sheets

Interer · Oct 30, 2023

Have you tried to clean it programmatically?

BlogPro · Oct 30, 2023

I wrote a similarity checking script.

If you set the similarity threshold in the script to 1 - it will remove exact duplicates - which is what you need.

If you know how to run a python script and your computer has decent power, a million lines shouldn't take long.

https://www.blackhatworld.com/seo/free-script-bulk-keyword-deduplication-similarity-check.1491191/
There is also a Google Colab version at the bottom of that post. You'll have to break down your list to smaller pieces to run it there though. Doubt they'll allocate enough resources for you to run several million lines worth.

Intofantasy · Oct 30, 2023

daverawcus said:
Google sheets

Would you mind explaning more?

Interer said:
Have you tried to clean it programmatically?

Could you please elaborate? I just used a program it's called notebook++ it has a feature that removes duplicates but I'm getting the error as I mentioned above.

BlogPro said:
I wrote a similarity checking script.

If you set the similarity threshold in the script to 1 - it will remove exact duplicates - which is what you need.

If you know how to run a python script and your computer has decent power, a million lines shouldn't take long.

https://www.blackhatworld.com/seo/free-script-bulk-keyword-deduplication-similarity-check.1491191/
There is also a Google Colab version at the bottom of that post. You'll have to break down your list to smaller pieces to run it there though. Doubt they'll allocate enough resources for you to run several million lines worth.

I will read the link you sent, and try my best.
Very frustrating shit, we live in an era of ai and still this cannot be done in simply task.

Interer · Oct 30, 2023

Intofantasy said:
Could you please elaborate? I just used a program it's called notebook++ it has a feature that removes duplicates but I'm getting the error as I mentioned above.

I can help you and clean it in Python if u wish

Intofantasy · Oct 30, 2023

Interer said:
I can help you and clean it in Python if u wish

I would appreciate, if this could be taught. I don't want to rely on too many sources.

Interer · Oct 30, 2023

If you have discord, I can explain it to you

BlogPro · Oct 30, 2023

Intofantasy said:
I will read the link you sent, and try my best.
Very frustrating shit, we live in an era of ai and still this cannot be done in simply task.

It's got nothing to do with AI or technological advancements. And everything to do with resources available - mostly hardware in this case.

Let's try to debug it

1. Can you try with a smaller file? Take your existing file and copy the first 200k or 500k lines to a new file.

Now run the dedupe utility. See if it runs through successfully.

If it does, it's a hardware limitation and the simplest method would be to break your file into 5-6 smaller files and run dedupe on each.

If it doesn't, then...

2. Can you paste your debug info here? You can access it by going to

? -> Debug info -> Copy debug info to clipboard

Interer · Oct 30, 2023

It mainly depends on how you have your data formatted, but basics of Python should be enough

Intofantasy · Oct 30, 2023

BlogPro said:
It's got nothing to do with AI or technological advancements. And everything to do with resources available - mostly hardware in this case.

Let's try to debug it

1. Can you try with a smaller file? Take your existing file and copy the first 200k or 500k lines to a new file.

Now run the dedupe utility. See if it runs through successfully.

If it does, it's a hardware limitation and the simplest method would be to break your file into 5-6 smaller files and run dedupe on each.

If it doesn't, then...

2. Can you paste your debug info here? You can access it by going to

? -> Debug info -> Copy debug info to clipboard

Thank you so much mate I'll try that immediately!

sageshark · Oct 30, 2023

@Intofantasy I have just merged 3 of your posts you made within short time interval replying different people. Ideally you should have used a multi-quotes feature to reply more than one person as a time. Here is a guide for you - https://www.blackhatworld.com/seo/how-to-use-the-multiquote-feature-on-bhw.975670/

Please note that failing to do so may attract warning for post count inflation. Hope for your cooperation.

lewwy789 · Oct 30, 2023

WPS, excel or Google sheets should be able to sort the dupes out.

Google sheets: paste sites in a sheet and highlight that column > click data on the top menu > data clean up > remove dupes

WPS: same as Google sheets paste sites in a sheet and highlight that column > click data on the top menu > highlight dupes > remove dupes

Excel: same again, paste sites in a sheet and highlight that column > click data on the top menu > remove dupes

Might be slow loading though with such a high number of sites.

TZ2011 · Oct 30, 2023

Try this: https://www.scrapebox.com/free-dupe-remove

Intofantasy · Oct 30, 2023

TZ2011 said:
Try this: https://www.scrapebox.com/free-dupe-remove

Thank you so much I'll definitely try this!

itz_styx · Oct 30, 2023

just use linux:

cat list.txt | sort -fu > sorted.txt

dandan594594 · Oct 30, 2023

A file splitter should help with first stage... Pinetools.com/split-files

Mark Eternally · Oct 30, 2023

I would probably first try to figure out why Notepad++ is not working properly, but while I was trying to figure that out, just to get the results I would try other options.

@itz_styx just posted a beautiful solution! Super simple to use! Not sure why my mind did not go this way. No need for additional applications or downloads. I was going to give a MS-DOS recommendation, but that linux recommendation is so on point that is the best response!

Some people above have mentioned this, but be sure the file type and character set are as generic as possible. I have had issues with odd character sets. Basically, sanitize your data before working it.

Anyhow, now that I see the simple fix posted by @itz_styx I would probably go that route.

Intofantasy · Oct 30, 2023

This really worked!

speedie said:
Search the forum, I'm sure you

Intofantasy said:
Create account and place order for what exacly you offer smm services not payment gateway accounts.
So stop spamming ur ads everywhere.

swp said:
yes, upload it and ask for the prompt . reverse engine technique.

Ashk881 said:
Is that the original image?

Several stable diffusion frontends like Dreamfusion and Automattic111 embed the prompt in the png metadata of the image itself.
So checking that should be a good place to start.

Otherwise you can use a CLIP-Interrogator tool. Here's one:
https://huggingface.co/spaces/pharmapsychotic/CLIP-Interrogator

nakamura said:
Yes, you can use Midjourney to describe the image and get a prompt. Then just use the same prompt to generate an image.

BlogPro said:
I wrote a similarity checking script.

If you set the similarity threshold in the script to 1 - it will remove exact duplicates - which is what you need.

If you know how to run a python script and your computer has decent power, a million lines shouldn't take long.

https://www.blackhatworld.com/seo/free-script-bulk-keyword-deduplication-similarity-check.1491191/
There is also a Google Colab version at the bottom of that post. You'll have to break down your list to smaller pieces to run it there though. Doubt they'll allocate enough resources for you to run several million lines worth.

Mark Eternally said:
I would probably first try to figure out why Notepad++ is not working properly, but while I was trying to figure that out, just to get the results I would try other options.

@itz_styx just posted a beautiful solution! Super simple to use! Not sure why my mind did not go this way. No need for additional applications or downloads. I was going to give a MS-DOS recommendation, but that linux recommendation is so on point that is the best response!

Some people above have mentioned this, but be sure the file type and character set are as generic as possible. I have had issues with odd character sets. Basically, sanitize your data before working it.

Anyhow, now that I see the simple fix posted by @itz_styx I would probably go that route.

This really worked!
Try this: https://www.scrapebox.com/free-dupe-remove
thank u!!!

Permat · Oct 30, 2023

Yes, Scrapebox can do this I think.

Remove Duplicates 2.5M+ LINES MORE! (HELP)! NOTEBOOK ++

Senior Member

Elite Member

Newbie

Elite Member

Senior Member

Newbie

Senior Member

Newbie

Elite Member

Newbie

Senior Member

Marketplace Moderator

Senior Member

Senior Member

Senior Member

Elite Member

Elite Member

Regular Member

Senior Member

Regular Member

Main Menu

Marketplace

Making Money

BlackHat World