[GET] Perl script for duplicated domains removal from text file (get unique domains)

xrfanatic

BANNED
Joined
Aug 28, 2010
Messages
466
Reaction score
201
Hi BHW,

since a vast majority of us work with loads of urls daily I'm sharing this little but very handy script which allows to remove duplicated domains from text file and get the only unique ones under Windows command line.

Getting unique domains from 100 mb file takes about 60 seconds, there are probably faster solutions, but this one is handy because it's standalone and you do not have to interfere with your seo software to take care about that while it's busy.

Usage in Windows cmd:

perl remduplicates.pl DuplicatedDomains.txt UniqueDomains.txt

Where:

DuplicatedDomains.txt - Your file with duplicated domains
UniqueDomains.txt - New file which will contain only unique domains after script's work is finished.

If you run this script on Windows, you need to install Strawberry Perl.

Code:
http://www.speedyshare.com/FARp2/remduplicates.pl

Hope it was useful :-)

Cheers !
 
Free shares like this gets little to no attention here in BHW for some reason. Even I too have shared some nice little hacks before, non cares, but are ready to buy the same functionality for dollars. Its good though. :D

Good to find a perl guy! :D
 
In Linux, scrubbing a file of duplicate entries is as simple as this one line:
Code:
awk '!x[$0]++' list.txt > cleaned.txt
 
Back
Top