How to remove duplicate domains from text files using python?

python remove_duplicate_domains.py input.txt output.txt --toplevel=true

import tldextract
import argparse


def rm_dup_domains(args):
    file_in = open(args.input, 'r', encoding='utf-8', errors='ignore')
    file_out = open(args.output, 'a', encoding='utf-8', errors='ignore')
    table = dict()

    for line in file_in:
        url = tldextract.extract(line.strip())

        if args.toplevel is True:
            domain = url.registered_domain

        elif args.toplevel is False:
            if url.subdomain == 'www' or url.subdomain == '':
                domain = url.registered_domain

            else:
                domain = url.fqdn
                if domain.startswith('www.'):
                    domain = url.fqdn.replace('www.', '', 1)

        if domain != '':
            # lookup for duplicates
            if table.get(domain) is not None:
                pass
            else:
                table[domain] = True
                file_out.write(f'{domain}\n')


if __name__ == '__main__':
    # create command line arguments
    parser = argparse.ArgumentParser()
    parser.add_argument('input', help='input file', type=str)
    parser.add_argument('output', help='output file', type=str)
    parser.add_argument('--toplevel', help='keeps only unique top-level domains (removes sub-domains)', type=bool, default=False)
    parser.parse_args()
    # parse them
    args = parser.parse_args()
    # run code
    rm_dup_domains(args)

Your post in the thread How to remove duplicate domains from text files using python? was deleted. Reason: Thread is from 2015, no need to bump it now
3 minutes ago

Click to expand...

ScrapeboxWorker said:

Yes very clever moderator... i won't use real epithets that i think what are you...
Now beacuse you deleted my reply, you have:
1. A duplicate thread... great!
2. Old topic was indexed in google and could drive traffic to this website... people looking for "how to remove duplicate domains in python" were coming to this forum and registering... now you have a fresh one that will take ages to index like the old one... great!
3. Instead of keep things in one topic, we will have 22340980 different threads...

Click to expand...

WilhelmScream said:

Actually, that was exactly the intention the moderator had behind warning you for necro-bumping - we'd rather have fresh threads created on topics instead of old, out-of-date ones being bumped instead of this.

If you'd have posted this without the passive-aggressive messages about having your post deleted, it would have been absolutely fine.

Click to expand...

bartosimpsonio said:

On linux all you gotta do is:

Code:

cat FILE.txt | sort | uniq

Click to expand...

ScrapeboxWorker said:

Why i can't see who deleted my post? In the alert i received there was no nick of the moderator ? Anonymous aha ?

Second in google... nice source of traffic... nice place to put usefull info for people to be found... if you think that creating a new thread is good for the forum and members itself then tell me why?
Tell me what is logic behind this?

Click to expand...

ScrapeboxWorker said:

What about
http://domain1.com/1
http://domain1.com/2

Click to expand...

cat FILE.txt | perl -pne 's/http[s]*\:\/\/([^\/]+)[\/]*.*/$1/' | sort | uniq

bartosimpsonio said:

You hadn't specified they were URLs and not domains! In that case it's:

Code:

cat FILE.txt | perl -pne 's/http[s]*\:\/\/([^\/]+)[\/]*.*/$1/' | sort | uniq

Works for both domains and urls.

Click to expand...

Asif WILSON Khan said:

Don't panic OP, your thread is Indexed

https://encrypted.google.com/search?q=How+to+remove+duplicate+domains+from+text+files+using+python?

Click to expand...

ScrapeboxWorker said:

Glad you could get your signature in this thread... still would be better to have in the old thread i guess.

Click to expand...