1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Need a tool that will delete duplicate domains

Discussion in 'Black Hat SEO' started by hellomotow07, Aug 27, 2010.

  1. hellomotow07

    hellomotow07 Power Member Premium Member

    Joined:
    Aug 24, 2010
    Messages:
    643
    Likes Received:
    350
    Ok, so i have been using ScrapeBox and have a big list of domains.

    Are there any programs out there that will delete duplicate domains (NOT URLS) just like the "Remove duplicate domains" button in ScrapeBox. It needs to be able to import a list of text files and of course keep 1 copy of the domain (not completely delete it)
     
  2. keith

    keith Junior Member

    Joined:
    Jan 26, 2010
    Messages:
    177
    Likes Received:
    127
    Occupation:
    Web Development & SEO
    Location:
    Outside of Chicago
    Home Page:
    What's wrong with Scrapebox's function? You can import text files into it and remove duplicates, then save back to text file pretty quickly.

    Or did I miss something?
     
  3. hellomotow07

    hellomotow07 Power Member Premium Member

    Joined:
    Aug 24, 2010
    Messages:
    643
    Likes Received:
    350
    I have over 100 million urls, that will take a hours not to mention SB has a limit of 1 million so after i get 1 million Unique domains i wont be able to import/remove duplicate
     
  4. hackNstuff

    hackNstuff BANNED BANNED Premium Member

    Joined:
    Jun 10, 2010
    Messages:
    136
    Likes Received:
    15
    I can throw together something in perl if you'd like, it would be a good exercise for me, interested? If you and a couple others want just post in this thread and I'll see if I can come up with something to post here in the next week or so...
     
  5. hellomotow07

    hellomotow07 Power Member Premium Member

    Joined:
    Aug 24, 2010
    Messages:
    643
    Likes Received:
    350
    That would be great. Right now im using Batch KeyWord Cleaner but that only removes duplicate URLS not domains.

    /blackhat-seo/member-downloads/145457-get-batch-keyword-cleaner-great-manipulating-url-keyword-lists.html#post1319388
     
  6. rabbitgirl

    rabbitgirl Regular Member

    Joined:
    May 21, 2010
    Messages:
    292
    Likes Received:
    233
    Location:
    Somewhere in the FuTuRe!
    100million Urls? Then Do 1million Urls 100-times [should be done within an hour]
     
  7. hellomotow07

    hellomotow07 Power Member Premium Member

    Joined:
    Aug 24, 2010
    Messages:
    643
    Likes Received:
    350
    Would take like 2 hours of doing boring tedious work (and ill be getting even more, so if there are any programs out there that can do this it would save me lots of time) and even then they wont all be unique. I can only import upto 1 million domains into SB, after about 10-15 million urls ill have that many and wont be able to import anymore.