1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

[Q] Proxy list cleaning, how to filter out identical ips with different port number?

Discussion in 'Proxies' started by HoNeYBiRD, Jul 14, 2014.

  1. HoNeYBiRD

    HoNeYBiRD Jr. VIP Jr. VIP

    Joined:
    May 1, 2009
    Messages:
    5,913
    Likes Received:
    7,150
    Gender:
    Male
    Occupation:
    Geographer, Tourism Manager
    Location:
    Ghosted
    How to filter out those proxies of a huge ass list, where the ip is identical, but the port number is different?

    Let's say i have list like this:

    HTML:
    1.11.11.111:80
    1.11.11.111:8080
    1.11.11.111:18799
    1.12.12.112.8080
    1.23.45.1223:3127
    201.12.13.4:3128
    201.12.13.4:8088
    and i want this as result:

    HTML:
    1.11.11.111:80
    1.12.12.112.8080
    1.23.45.1223:3127
    201.12.13.4:8088
    How can i do this quickly and effectively w/o the need of testing the list?

    At this stage i don't care about port numbers, if they're bad or good, i just want the duplicate ips to be removed and keep only one from each duplicate.

    I know Charon can do this, but it takes ages until it imports a huge list, so is there any quicker?
    It can be a simple text editing tool, notepad++ or excel, doesn't need to be a proxy tool.
    I just can't seem to wrap my head around this.

    Thanks
     
  2. ronegraT

    ronegraT Power Member

    Joined:
    Dec 29, 2010
    Messages:
    620
    Likes Received:
    101
    Occupation:
    sleeping
    Location:
    Sweden
    I used scrapebox and the function remove duplicate domains

    which made this
    into this

    Only extra it does is that its adding http:// , but thats easy to remove in a text editor later on
     
    • Thanks Thanks x 1
  3. akacash

    akacash Jr. VIP Jr. VIP

    Joined:
    Jan 16, 2010
    Messages:
    806
    Likes Received:
    575
    Location:
    The Beach, USA
    • Thanks Thanks x 1
  4. HoNeYBiRD

    HoNeYBiRD Jr. VIP Jr. VIP

    Joined:
    May 1, 2009
    Messages:
    5,913
    Likes Received:
    7,150
    Gender:
    Male
    Occupation:
    Geographer, Tourism Manager
    Location:
    Ghosted
    great, that's exactly what i need, quick and simple
    i have the free dupe remove tool myself since ages, i have only used it what's it's for, but i haven't thought about this use until now

    i guess 2 solution is even better than one :)