Hi BHW,
since a vast majority of us work with loads of urls daily I'm sharing this little but very handy script which allows to remove duplicated domains from text file and get the only unique ones under Windows command line.
Getting unique domains from 100 mb file takes about 60 seconds, there are probably faster solutions, but this one is handy because it's standalone and you do not have to interfere with your seo software to take care about that while it's busy.
Usage in Windows cmd:
perl remduplicates.pl DuplicatedDomains.txt UniqueDomains.txt
Where:
DuplicatedDomains.txt - Your file with duplicated domains
UniqueDomains.txt - New file which will contain only unique domains after script's work is finished.
If you run this script on Windows, you need to install Strawberry Perl.
Hope it was useful
Cheers !
since a vast majority of us work with loads of urls daily I'm sharing this little but very handy script which allows to remove duplicated domains from text file and get the only unique ones under Windows command line.
Getting unique domains from 100 mb file takes about 60 seconds, there are probably faster solutions, but this one is handy because it's standalone and you do not have to interfere with your seo software to take care about that while it's busy.
Usage in Windows cmd:
perl remduplicates.pl DuplicatedDomains.txt UniqueDomains.txt
Where:
DuplicatedDomains.txt - Your file with duplicated domains
UniqueDomains.txt - New file which will contain only unique domains after script's work is finished.
If you run this script on Windows, you need to install Strawberry Perl.
Code:
http://www.speedyshare.com/FARp2/remduplicates.pl
Hope it was useful
Cheers !