Remove Duplicates 2.5M+ LINES MORE! (HELP)! NOTEBOOK ++

[Intofantasy]

Is there a software out there, that can handle more than 2.5 million duplicates?
I'm using notebook ++, I'm trying to remove duplicate lines, in some cases I'm going over 5 or 7 million lines.

I just want to Simply remove the duplicate lines so the list becomes more efficient and smaller.

When I'm using notebook ++, I'm getting error messages " bad allocation" and everything just closes and errors.

 

[daverawcus]

Google sheets

 

[Interer]

Have you tried to clean it programmatically?

 

[BlogPro]

I wrote a similarity checking script.

If you set the similarity threshold in the script to 1 - it will remove exact duplicates - which is what you need.

If you know how to run a python script and your computer has decent power, a million lines shouldn't take long.

https://www.blackhatworld.com/seo/free-script-bulk-keyword-deduplication-similarity-check.1491191/
There is also a Google Colab version at the bottom of that post. You'll have to break down your list to smaller pieces to run it there though. Doubt they'll allocate enough resources for you to run several million lines worth.

 

[Intofantasy]

Google sheets

Would you mind explaning more?

Have you tried to clean it programmatically?

Could you please elaborate? I just used a program it's called notebook++ it has a feature that removes duplicates but I'm getting the error as I mentioned above.

I wrote a similarity checking script.

If you set the similarity threshold in the script to 1 - it will remove exact duplicates - which is what you need.

If you know how to run a python script and your computer has decent power, a million lines shouldn't take long.

https://www.blackhatworld.com/seo/free-script-bulk-keyword-deduplication-similarity-check.1491191/
There is also a Google Colab version at the bottom of that post. You'll have to break down your list to smaller pieces to run it there though. Doubt they'll allocate enough resources for you to run several million lines worth.

I will read the link you sent, and try my best.
Very frustrating shit, we live in an era of ai and still this cannot be done in simply task.

 

[Interer]

Could you please elaborate? I just used a program it's called notebook++ it has a feature that removes duplicates but I'm getting the error as I mentioned above.

I can help you and clean it in Python if u wish

 

[Intofantasy]

I can help you and clean it in Python if u wish

I would appreciate, if this could be taught. I don't want to rely on too many sources.

 

[Interer]

If you have discord, I can explain it to you

 

[BlogPro]

I will read the link you sent, and try my best.
Very frustrating shit, we live in an era of ai and still this cannot be done in simply task.

It's got nothing to do with AI or technological advancements. And everything to do with resources available - mostly hardware in this case.

Let's try to debug it

1. Can you try with a smaller file? Take your existing file and copy the first 200k or 500k lines to a new file.

Now run the dedupe utility. See if it runs through successfully.

If it does, it's a hardware limitation and the simplest method would be to break your file into 5-6 smaller files and run dedupe on each.

If it doesn't, then...

2. Can you paste your debug info here? You can access it by going to

? -> Debug info -> Copy debug info to clipboard

 

[Interer]

It mainly depends on how you have your data formatted, but basics of Python should be enough

 

[Intofantasy]

It's got nothing to do with AI or technological advancements. And everything to do with resources available - mostly hardware in this case.

Let's try to debug it

1. Can you try with a smaller file? Take your existing file and copy the first 200k or 500k lines to a new file.

Now run the dedupe utility. See if it runs through successfully.

If it does, it's a hardware limitation and the simplest method would be to break your file into 5-6 smaller files and run dedupe on each.

If it doesn't, then...

2. Can you paste your debug info here? You can access it by going to

? -> Debug info -> Copy debug info to clipboard

Thank you so much mate I'll try that immediately!

 

[sageshark]

@Intofantasy I have just merged 3 of your posts you made within short time interval replying different people. Ideally you should have used a multi-quotes feature to reply more than one person as a time. Here is a guide for you - https://www.blackhatworld.com/seo/how-to-use-the-multiquote-feature-on-bhw.975670/

Please note that failing to do so may attract warning for post count inflation. Hope for your cooperation.

 

[lewwy789]

WPS, excel or Google sheets should be able to sort the dupes out.

Google sheets: paste sites in a sheet and highlight that column > click data on the top menu > data clean up > remove dupes

WPS: same as Google sheets paste sites in a sheet and highlight that column > click data on the top menu > highlight dupes > remove dupes

Excel: same again, paste sites in a sheet and highlight that column > click data on the top menu > remove dupes

Might be slow loading though with such a high number of sites.

 

[TZ2011]

Try this: https://www.scrapebox.com/free-dupe-remove

 

[Intofantasy]

Try this: https://www.scrapebox.com/free-dupe-remove

Thank you so much I'll definitely try this!

 

[itz_styx]

just use linux:

cat list.txt | sort -fu > sorted.txt

 

[dandan594594]

A file splitter should help with first stage... Pinetools.com/split-files

 

[Mark Eternally]

I would probably first try to figure out why Notepad++ is not working properly, but while I was trying to figure that out, just to get the results I would try other options.

@itz_styx just posted a beautiful solution! Super simple to use! Not sure why my mind did not go this way. No need for additional applications or downloads. I was going to give a MS-DOS recommendation, but that linux recommendation is so on point that is the best response!

Some people above have mentioned this, but be sure the file type and character set are as generic as possible. I have had issues with odd character sets. Basically, sanitize your data before working it.

Anyhow, now that I see the simple fix posted by @itz_styx I would probably go that route.

 

[Intofantasy]

This really worked!

Search the forum, I'm sure you

Create account and place order for what exacly you offer smm services not payment gateway accounts.
So stop spamming ur ads everywhere.

yes, upload it and ask for the prompt . reverse engine technique.

Is that the original image?

Several stable diffusion frontends like Dreamfusion and Automattic111 embed the prompt in the png metadata of the image itself.
So checking that should be a good place to start.

Otherwise you can use a CLIP-Interrogator tool. Here's one:
https://huggingface.co/spaces/pharmapsychotic/CLIP-Interrogator

Yes, you can use Midjourney to describe the image and get a prompt. Then just use the same prompt to generate an image.

I wrote a similarity checking script.

If you set the similarity threshold in the script to 1 - it will remove exact duplicates - which is what you need.

If you know how to run a python script and your computer has decent power, a million lines shouldn't take long.

https://www.blackhatworld.com/seo/free-script-bulk-keyword-deduplication-similarity-check.1491191/
There is also a Google Colab version at the bottom of that post. You'll have to break down your list to smaller pieces to run it there though. Doubt they'll allocate enough resources for you to run several million lines worth.

I would probably first try to figure out why Notepad++ is not working properly, but while I was trying to figure that out, just to get the results I would try other options.

@itz_styx just posted a beautiful solution! Super simple to use! Not sure why my mind did not go this way. No need for additional applications or downloads. I was going to give a MS-DOS recommendation, but that linux recommendation is so on point that is the best response!

Some people above have mentioned this, but be sure the file type and character set are as generic as possible. I have had issues with odd character sets. Basically, sanitize your data before working it.

Anyhow, now that I see the simple fix posted by @itz_styx I would probably go that route.

This really worked!
Try this: https://www.scrapebox.com/free-dupe-remove
thank u!!!

 

[Permat]

Yes, Scrapebox can do this I think.