Can't Delete Duplicates 25 Million URL List Every Program Crashes Xrumer / gScraper notepa

[smokemeoutdawg]

I got a 25 million list I harvested using gScraper and I am trying to delete duplicate domains, easier said than done with a 4gb file

I attempted to delete the duplicates with gScraper and it crashes after taking up 14GB of ram lol. Tried Xrumer and it gives a error. Too big.

It's too big for Notepad ++ to open fully.

Tried splitting it with gSplit but it's too big for that too.

I need a easy way to split big files if anyone could recommend anything that would be great.

 

[babasss]

try scrapebox dupe remover

 

[tahworld]

Hey try this, it was made specifically for big text files:

It supposed to be able to handle into the Billions.

http://www.emeditor.com/

 

[Winternacht]

Scrapebox Dupe Remove
http://www.scrapebox.com/free-dupe-remove

 

[PrinceVisi]

Can't you split them into 25 files with 1 million each and retest?

 

[smokemeoutdawg]

try scrapebox dupe remover

Tried that too. Lag for hours. No success.

Hey try this, it was made specifically for big text files:

It supposed to be able to handle into the Billions.

http://www.emeditor.com/

Heisenberg for the win! This actually opened up my file in seconds, amazing man.

Are you familiar with how to remove dups using this program? Or Spilt into 2?

 

[davids355]

I have a server with 128gb ram
could give it a go if you want, but probably take me a few days.

 

[skadster]

You could also try editpad lite, I'm sure it has a command to delete duplicate lines.

 

[d2ugsd]

I got a 25 million list I harvested using gScraper and I am trying to delete duplicate domains, easier said than done with a 4gb file

I attempted to delete the duplicates with gScraper and it crashes after taking up 14GB of ram lol. Tried Xrumer and it gives a error. Too big.

It's too big for Notepad ++ to open fully.

Tried splitting it with gSplit but it's too big for that too.

I need a easy way to split big files if anyone could recommend anything that would be great.

UltraEdit will do the job

 

[smokemeoutdawg]

I have a server with 128gb ram
could give it a go if you want, but probably take me a few days.

Its not the ram that's the problem, I got one with 32gb ram, it's the software.

So I ran it through Heisenburgs program and it spilts them but adds a " before and after the URL

This causes errors. Any other recommendations are highly appreciated.

You could also try editpad lite, I'm sure it has a command to delete duplicate lines.

Editpad does not support files larger than 2GB.


Any other recommendations are highly appreciated.

 

[d2ugsd]

I worked with 50gb file with ultraedit so 4gb is peace of cake.

 

[Groen]

Have you tried the scrapebox addon DupRemove as other users suggested? It should handle up to 180 million lines at a time.