1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

SB dupe remove addon take 24hrs+ to remove dupe URLs :p

Discussion in 'Black Hat SEO Tools' started by bk071, Apr 4, 2011.

  1. bk071

    bk071 Jr. Executive VIP Jr. VIP Premium Member

    Joined:
    Nov 24, 2010
    Messages:
    3,126
    Likes Received:
    7,926
    Occupation:
    I don't have a job
    Location:
    .............
    Yeah this may sound like crazy but it does :p

    I harvested around 14 million URLs and merged them in one Huge file. Then loaded that file in DupeRemove and pressed remove duplicate URLs. And there it is, over 24 hours and its still "writing unique URLs". No, I'm not on a low end PC :)

    Any other tools to do the job? I know loopline's but that trims to root domain when removing dupe domains.

    Anyone?
    Bk...
     
  2. bullseye123

    bullseye123 Regular Member

    Joined:
    May 4, 2010
    Messages:
    287
    Likes Received:
    126
    Occupation:
    IT Support
    Location:
    South Africa
    Don't think you will find something faster than SB here. 14 mil links, don't think you will even get a 1% success with that list.
     
  3. bk071

    bk071 Jr. Executive VIP Jr. VIP Premium Member

    Joined:
    Nov 24, 2010
    Messages:
    3,126
    Likes Received:
    7,926
    Occupation:
    I don't have a job
    Location:
    .............
    • Thanks Thanks x 1
  4. -FPC-

    -FPC- Regular Member

    Joined:
    Apr 1, 2011
    Messages:
    341
    Likes Received:
    68
    Occupation:
    Professional freelance journalist, researcher, aut
    Location:
    Southern California
    It is actually a pretty good list. Would you be willing to share your footprint? :D
     
  5. -Jericho-

    -Jericho- Jr. Executive VIP

    Joined:
    Jan 10, 2010
    Messages:
    2,849
    Likes Received:
    1,704
    Location:
    Stalking My Ex-Wife
    Crazyflx has a method on his website for removing duplicates that's much easier. I forget which programs he suggests to use. Send him a PM. I'm sure he can tell you how he does it.
     
  6. philionaire

    philionaire Regular Member

    Joined:
    Mar 20, 2010
    Messages:
    212
    Likes Received:
    180
    Location:
    Vanland
    You could try this:

    Code:
    http://www.bigbangenterprises.de/en/doublekiller/
    Ive used it before and it works fine. Dont know time wise for yours though!

    Theres also 1 text pipe pro on here:

    Code:
    http://www.blackhatworld.com/blackhat-seo/black-hat-seo-tools/276208-textpipe-pro-8-6-7-best-text-manipulation-tool-ever.html
    HTH.
     
  7. Sinatra

    Sinatra Junior Member

    Joined:
    Feb 28, 2011
    Messages:
    128
    Likes Received:
    30
    Location:
    Canada
    Hey, I just checked crazyflx's blog, and he recommends the duperemove tool too which bk is using. Of course he put something else I missed.

    Anyhow, keep up the good shares bk, thanks given on other thread.
     
  8. freller

    freller Regular Member

    Joined:
    Sep 26, 2008
    Messages:
    210
    Likes Received:
    65
    14 million URLs in one file is too big - you need to split it into 1 million URL files and then go from there.