1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Where Do Sites Like ExpiredDomains.net / Moonsy Get Their Lists From?

Discussion in 'Black Hat SEO' started by MrBeastsOnToast, Jun 11, 2014.

  1. MrBeastsOnToast

    MrBeastsOnToast Jr. VIP Jr. VIP Premium Member

    Joined:
    Dec 17, 2011
    Messages:
    921
    Likes Received:
    537
    Location:
    The Internetz
    Hi Guys,

    Whoever owns these sites is clearly scraping the best picks and keeping them for themselves while publishing the rest for ad revenue from us. Anyone know where they scrape these lists from? Do they literally crawl millions of domains looking for dead links?

    Im an experienced coder and going to code my own bot, but not too sure how they even do it.

    Thanks!
     
  2. DamageX

    DamageX Elite Member

    Joined:
    Sep 17, 2008
    Messages:
    2,692
    Likes Received:
    1,687
    Occupation:
    Unemployable
    Location:
    Former nomad
    OT, what kind of coding do you do? PHP + MySQL, Windows apps, both? I'm asking because I'm looking for a coder to JV with, to implement an plan I've been working on for some time now.
     
  3. Kralj187

    Kralj187 Senior Member

    Joined:
    Jul 9, 2012
    Messages:
    1,046
    Likes Received:
    222
    Location:
    Internet
    I think those who know won't say it here...
     
    • Thanks Thanks x 5
  4. MrBeastsOnToast

    MrBeastsOnToast Jr. VIP Jr. VIP Premium Member

    Joined:
    Dec 17, 2011
    Messages:
    921
    Likes Received:
    537
    Location:
    The Internetz
    I thought as much... Don't suppose a free copy of the bot would entice a PM to my inbox? (bot wont be for sale).
     
  5. MrBeastsOnToast

    MrBeastsOnToast Jr. VIP Jr. VIP Premium Member

    Joined:
    Dec 17, 2011
    Messages:
    921
    Likes Received:
    537
    Location:
    The Internetz
    PM Sent dude. 15 chars
     
  6. ikilledkenny

    ikilledkenny Junior Member

    Joined:
    May 1, 2012
    Messages:
    171
    Likes Received:
    62
    i'm not a technical guy but since moonsy (and i think also the other one, I don't use it) got only godaddy, then i guess its scraping these lists lol? for other sites scrape their lists too. either way - no one scraping the entire net looking for dead links bro, that's just plain stupid
     
  7. MrBeastsOnToast

    MrBeastsOnToast Jr. VIP Jr. VIP Premium Member

    Joined:
    Dec 17, 2011
    Messages:
    921
    Likes Received:
    537
    Location:
    The Internetz
    Nevermind, found it out.
     
    Last edited: Jun 11, 2014
  8. jackl

    jackl Regular Member

    Joined:
    Apr 20, 2014
    Messages:
    218
    Likes Received:
    39
    Mind sharing with us?
     
  9. MrBeastsOnToast

    MrBeastsOnToast Jr. VIP Jr. VIP Premium Member

    Joined:
    Dec 17, 2011
    Messages:
    921
    Likes Received:
    537
    Location:
    The Internetz
    Been told not to by the person that shared with me otherwise i would. Sorry.
     
  10. freidagulal

    freidagulal Newbie

    Joined:
    Apr 29, 2014
    Messages:
    10
    Likes Received:
    1
    Multiple sources I would suppose, these sites do a good job though domains listed as available do not always prove as such.
     
  11. prab1996

    prab1996 Elite Member

    Joined:
    Jan 8, 2013
    Messages:
    3,496
    Likes Received:
    2,028
    Occupation:
    your gf's <3 ♥♥♥♥
    Location:
    Prab1996.com
    Home Page:
    they just scrap godaddy auctions and other domain auction sites. for some gold nuggets you will have to do scraping yourself.
    -=-
     
  12. axemantech

    axemantech Newbie

    Joined:
    Sep 12, 2013
    Messages:
    40
    Likes Received:
    25
    I own and run DomCop. I can tell you how we get the list of our domains. We get them from the following sources.

    1. GoDaddy domain list over FTP
    2. SnapNames Pre-release and Pending Delete Lists
    3. NameJet Pending Delete lists
    4. DynaDot Pending Delete lists
    5. Sedo Pending Delete lists
    6. Multiple drop lists available for different foreign tlds

    We process about 300,000 domains a day. Getting the domains is actually not such a big issue. A bigger issue to get various metrics for the domains and to ensure the search works fast (we have around 12 million available domains in our archived list)

    If you have any more questions on how the site works, do let me know. I'd be happy to answer in any way that I can.
     
    • Thanks Thanks x 8
  13. Asif WILSON Khan

    Asif WILSON Khan Executive VIP Premium Member

    Joined:
    Nov 10, 2012
    Messages:
    10,139
    Likes Received:
    28,607
    Gender:
    Male
    Occupation:
    Fun Lovin' Criminal
    Location:
    London
    Home Page:
    Feel free to expand on that, might be interesting.
     
  14. axemantech

    axemantech Newbie

    Joined:
    Sep 12, 2013
    Messages:
    40
    Likes Received:
    25
    "A bigger issue to get various metrics for the domains"

    DomCop gets about 40 different metrics for every domain (Page Rank, SEMrush, Majestic Seo, Similar Web, Moz etc). All of these come from different APIs. Technically, this means having multiple scripts running concurrently on the server updating these for all domains. Processing 300,000+ domains everyday means that we need to update 3.4 domains with all the metrics every single second.

    Getting the domains in the database is a simple import script and pretty easy to create and setup. Making sure the server constantly updates the domains and still gives a fast enough response to user queries is the challenge. (for anyone looking at building something similar)
     
    • Thanks Thanks x 1
  15. ZennoBlaster

    ZennoBlaster Senior Member

    Joined:
    Jan 17, 2014
    Messages:
    1,025
    Likes Received:
    306
    In your own personal opinion, if you were to buy a domain (for, say, a PBN), which metrics would make you press the buy it now button? I hoping to automate this and use APIs to pull in relevant metrics.
     
  16. theIMMachine

    theIMMachine Regular Member

    Joined:
    Jul 2, 2011
    Messages:
    339
    Likes Received:
    138
    Gender:
    Male
    Occupation:
    IM, Designs
    Location:
    Everywhere :)
    1. How do you check the PR's and display it?
    2. What about checking the Alexa, only with Dmoz Entry , etc

    I mean we can filter and find the domains by sorting out with these metrics on Expireddomains.

    Would be great if you could reveal how they do it.
     
  17. axemantech

    axemantech Newbie

    Joined:
    Sep 12, 2013
    Messages:
    40
    Likes Received:
    25
    To buy a domain for a PBN I would check the following
    1. Ratio of Citation Flow to Trust Flow ( should be close to 1)
    2. Spam check on Anchor Texts to filter out previously PBNed/SEOed domains (at DomCop we are working at automating this)
    3. Value of Citation & Trust Flow
    4. SEMrush and SimilarWeb Ranks/Traffic stats
    5. Quality and number of back links (manual check required here)

    This is only my humble personal opinion. I would not consider myself an expert on PBNs. DomCop has many users who prefer using a different set of metrics to this and have done so successfully.
     
    • Thanks Thanks x 3
  18. axemantech

    axemantech Newbie

    Joined:
    Sep 12, 2013
    Messages:
    40
    Likes Received:
    25
    I cannot tell you how Expireddomains does it. But I can tell you how we do it at DomCop.

    We used to get the PR directly via a script we had written, but it was too much of a hassle to maintain. Now, we pay for an API that gets us the PRs. However, we verify the PRs ourselves by running an info: domain. In the future we plan to also check the wayback archive 301 redirects to verify the PR.

    Regarding Alexa and Dmoz you can use the Alexa API to get the data directly.
     
  19. royserpa

    royserpa Jr. VIP Jr. VIP Premium Member

    Joined:
    Sep 28, 2011
    Messages:
    4,655
    Likes Received:
    3,498
    Gender:
    Male
    Occupation:
    Negative Options aka Rebills!
    Location:
    Royserpa
    Home Page:
    Thats pretty cool, axem.

    Suscribed for later use.
     
  20. cashcorp

    cashcorp Regular Member

    Joined:
    Feb 8, 2008
    Messages:
    430
    Likes Received:
    270
    Home Page:
    Domains go to auction on numerous registrars, AFTER they "Drop" but before they are deleted. This is where the lists come from.

    If the auction does not sell, they transfer to deleted/instant reg status.

    To expand on the Moz and PR questions:

    Opensite/Moz has an API. Easily accessed programatically.

    Google Pagerank can be checked via the google toolbar. IE: http://toolbarqueries.google.com/

    Sample Code: (Cant vouch for this, My own is custom coded to work with my particular setup and would not be much use to someone though.)
    http://99webtools.com/php-script-to-get-pagerank.php

    If your serious about pulling metrics on domains, this library is extremely useful:
    https://github.com/eyecatchup/SEOstats
     
    Last edited: Jun 13, 2014