1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Help with large lists

Discussion in 'Black Hat SEO' started by apeagle, Oct 10, 2013.

  1. apeagle

    apeagle Newbie

    Joined:
    Oct 9, 2013
    Messages:
    15
    Likes Received:
    1
    Hello everyone! I am a long time lurker-first time poster but I can not seem to find out how to do this anywhere and I have been trying to find the answer to this for weeks! I know you guys are all smarter than me so here goes...

    I have one list of about 150,000 trimmed to root DOMAINS from pages that I have posted to that looks like this (for example):
    dogclothes.c.o.m.
    catshirts.c.o.m.

    I have a new list of URLS that I just got done scraping that is around 1 million that I am ready to post to that is setup like this:
    ...dogclothes.c.o.m/the-best-clothing-for-dogs/
    ...siteihaventpostedto.c.o.m/insert-page-here/
    ...catshirts.c.o.m/cats-rock

    I want to keep the 2nd result in the url list but lose the first and third result because I have already expanded and posted to the domains in my 1st list.

    What program/addon/plugin/ANYTHING would take my domains list and find AND delete the urls from the second list that have that domain in them, even if the domains in question are part of a longer url?