1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

SB Plugin: Dup Remove Sux Epic Balls !

Discussion in 'Black Hat SEO' started by HostStage, Aug 1, 2011.

  1. HostStage

    HostStage Jr. VIP Jr. VIP Premium Member UnGagged Attendee

    Joined:
    May 20, 2010
    Messages:
    1,773
    Likes Received:
    1,730
    Occupation:
    BHW - CEO of Webhosting Company
    Location:
    BWH from France
    Home Page:
    Duplicate Remove Plugin is meant to remove duplicate URL from Big list.

    But actually it is built to make you loose great amount of time.

    I've imported 1.3 Millions URL in it to remove duplicates.

    Well, it works like hell !

    I let him running 1 night with a I7 quadcore CPU and 6 gig of RAM.

    No progression bar or anything so after 12 hours i decided to terminate the processus.

    I open the list into SB (1 millions max) click remove duplicate 10 seconds later the job was done ....

    This thread is just a complaining one for the loss of time, i've been through while i was sleeping :D


    Is someone using it ?
     
  2. TheMatrix

    TheMatrix BANNED BANNED

    Joined:
    Dec 20, 2008
    Messages:
    3,444
    Likes Received:
    7,279
    I myself use the addon for managing very large lists (10 million+). For smaller ones, SB itself is enough...
     
    • Thanks Thanks x 1
  3. GoldenGlovez

    GoldenGlovez Moderator Staff Member Moderator Jr. VIP

    Joined:
    Mar 23, 2011
    Messages:
    701
    Likes Received:
    1,713
    Location:
    Guangdong, China
    Home Page:
    Are you running the latest version of the SB Dup Remover plugin? I've used the plugin numerous times for lists with 5+ Million URLs without a problem. Even on a machine with much lower specs than yours.
     
  4. HostStage

    HostStage Jr. VIP Jr. VIP Premium Member UnGagged Attendee

    Joined:
    May 20, 2010
    Messages:
    1,773
    Likes Received:
    1,730
    Occupation:
    BHW - CEO of Webhosting Company
    Location:
    BWH from France
    Home Page:
    oh, you do ?

    Because i have a lot to do too, but how long does it take you to remove dup from a 10 millions+ list ? a week ?

    @GoldenGlovez : I think so, i've downloaded it yesterday.

    With your answers, i think i know what is wrong :)

    I use inline URLs merged with 1 line 1 URL, could it be the problem ?
     
  5. GoldenGlovez

    GoldenGlovez Moderator Staff Member Moderator Jr. VIP

    Joined:
    Mar 23, 2011
    Messages:
    701
    Likes Received:
    1,713
    Location:
    Guangdong, China
    Home Page:
    I'm sure you will see issues if you are trying to compare lists/remove dups with length URL strings. Try using a simple URL list and see if that's the cause.
     
    • Thanks Thanks x 1