1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

anyone has a larger list of websites like 10million or more unique domains that are live

Discussion in 'Black Hat SEO' started by nanexo, Jun 3, 2015.

  1. nanexo

    nanexo BANNED BANNED

    Joined:
    Feb 14, 2010
    Messages:
    873
    Likes Received:
    188
    as the title says
    pretty much interested in any list or data if can be found online or perhaps someone has it
    a lot of you must be scrapers so maybe one of you has this data


    looking for any list of live websites
    dont care about the quality
    prefer no parked domains etc but actual live sites

    ideas on how to go about this with footprinting perhaps would also be an option but this would either require a lot of footprints or almost the main dictionary
    options options


    what did you guys do? again if anyone has the data please pm
     
  2. rogerke

    rogerke Regular Member

    Joined:
    Oct 5, 2014
    Messages:
    264
    Likes Received:
    145
    I have some lists containing millions of country specific ltd's including PR and some other metrics but those are for private use.

    Scraping them yourself is very easy. Just do several site:.ltd queries and use a shitload of keywords in multiple languages (FurryKyle on here sells an awesome list), let it scrape for a few days and you'll have your 10 million + list.
     
    • Thanks Thanks x 1
  3. nanexo

    nanexo BANNED BANNED

    Joined:
    Feb 14, 2010
    Messages:
    873
    Likes Received:
    188
    yea currently using that bandit scrapeapp and that has huge dictionaries
    can also diy with sb box
    but maybe anyone knows of some official resource or other resources that provide shortcuts over days of scraping is what im after
     
  4. Earned Results

    Earned Results Junior Member

    Joined:
    Jun 8, 2015
    Messages:
    104
    Likes Received:
    42
    2nd for the DIY Scrapebox approach.

    It will be more effective than a shared list of spammed/dead sites anyway since it's scraped from Google.
     
  5. tony_d

    tony_d Elite Member

    Joined:
    Jun 22, 2013
    Messages:
    2,583
    Likes Received:
    3,179
    Location:
    1600 Amphitheatre Parkway, Mountain View CA
    PM me if you don't find another solution, I can do an extract from my DB in a few hours.

    They won't all be live - it will be warts-n-all, but on average about 88-90% are live and currently registered.
     
    • Thanks Thanks x 1
  6. satyr85

    satyr85 Jr. VIP Jr. VIP

    Joined:
    Aug 7, 2011
    Messages:
    613
    Likes Received:
    468
    Location:
    Poland
    I have list of 32 milion domains and im preparing to make bigger list. Sending you PM.

    Awesome lists you say... Check index count for KWs in his list. You will be suprised.
     
    • Thanks Thanks x 1
    Last edited: Jun 8, 2015
  7. dotcomdeb

    dotcomdeb Newbie

    Joined:
    Mar 17, 2015
    Messages:
    11
    Likes Received:
    1
    Gender:
    Female
    Occupation:
    IMarketer
    Location:
    Canada
    Interesting......I am new to IM...why would you want a list of 32 million domains. Are they expired?
     
  8. rogerke

    rogerke Regular Member

    Joined:
    Oct 5, 2014
    Messages:
    264
    Likes Received:
    145
    Interesting. Never checked it, but I think it's still the most comprehensive list out there. At least I don't know about any better ones.
     
    • Thanks Thanks x 1
  9. PHPInjected

    PHPInjected Elite Member

    Joined:
    Apr 25, 2014
    Messages:
    2,148
    Likes Received:
    1,896
    Occupation:
    100% Unique Content Writer
    Location:
    Overriding Methods
    For future reference, you can scrape lists just like these with an imacros & javascript scipt based upon your screen coordinates. It is actually not that hard once you get into learning it.