1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

[Tutorial] How to Scrape for Expired Domains [12,223 Characters]

Discussion in 'Black Hat SEO' started by elavmunretea, May 8, 2017.

  1. elavmunretea

    elavmunretea Elite Member

    Joined:
    May 14, 2016
    Messages:
    1,580
    Likes Received:
    2,124
    Home Page:
    Long story short, I was writing an ebook to release on a few SEO forums, and when it came to re-drafting it to target a higher-tier of Marketers, I realised that's not the route that I want to go down. How can someone that knows so little about marketing release a book on the subject? Anyway... a section of the book was dedicated to Domains, and a portion of that was explaining the process I use to find expired domains. Here it is...

    ----------
    Saving Money with Expired Domains

    Domain scrapers are incredibly powerful tools that scour the web looking for expired domains with impressive metrics that webmasters have forgotten about/ given up on. I’ve been scraping domains for a while now and I’ve found more gold domains than I know what to do with.

    Even if you’re not interested in using an expired domain for your money site or PBN, you can make a sizeable bit of profit by flipping the domains – trust me.

    The Process - How it works:

    [​IMG]

    They typically work by taking a Keyword, finding the top 50 or so websites that rank for that search (In the country that you chose) and then scraping every single outbound link on those 50 sites. After finding all of the domains that are linked to by the related sites, they sort them by different metrics and hey presto, you now have a list of expired domains.

    The process isn’t the easiest thing to wrap your head around, and there’s no guarantee that you will even find a mediocre domain, but with the tips and tricks I’m going to come on to in a minute, you stand a good chance of making an amazing ROI on your subscription fee.

    What Keywords to use:

    As I mentioned above, the vast majority of Domain Scrapers use a Keyword to search

    When deciding on a keyword to search with, there are a few things that you should consider.

    If a keyword has almost no competition and most of the sites in the top 50 are local stores, linking out to 1-2 other sites, you’re not going to come across a whole lot of decent expired domains.

    On the other hand, if you search for something like “Payday loans” with hundreds of authoritative websites competing for the top rank, competition is going to be rife. There will be hundreds of other people searching for these highly authoritative terms and, as the saying goes, too many cooks spoil a broth. You’re not going to be able to find as many good domains because they’ll be snatched up right away.

    [​IMG]
    That’s why it’s important to find the middle-ground. Checking the top 50 results manually before scraping is a good idea as it ensures you’re not wasting your time on pointless searches. When inspecting the SERPs, make sure that there are at least a handful of authority sites that have a few thousand indexed pages.

    Metrics:

    I assume you all know what the various domain metrics are and what they represent (Have a read of @Nargil's explanation on the Current Expired Domain Situation) so I'm not going to go into too much detail here.

    One thing that's important to remember is that these metrics aren't the be-all and end-all of a domain.

    They're not official metrics released by Google. They're invented up by companies who, at the end of the day, profit from SEO’ers interest in rankings, which is why it’s not a good idea to place all of your trust in them.

    [​IMG]
    I'm not saying that these service providers literally pluck numbers out of thin air (I'm also not saying that none of them don't do this) I'm just reminding you that they're nothing official and that makes them easy to manipulate.
    Reffering Domains: If your site has 1000 backlinks, but only one referring domain, that’s not good news. Not only does this look incredibly dubious in the eyes of Google, but it also means you’ll lose 100% of your backlinks if that one site goes offline.

    While more referring domains doesn’t necessarily mean more security – you could have 100 beefy backlinks from one authority site and another 100 links from crappy blogs – it’s a good indication of quality.

    Different tools will return different scores for the same domain because they’ll all have slightly different indexes. PBNLab, my tool of choice, uses Majestic’s API.

    Domain Authority: When checking through the domains you’ve scraped, you will undoubtedly come across a handful of expired domains with ridiculously high domain authority and, nine times out of ten, there’s a reason that they haven’t been registered yet. Make sure to be extra careful when checking high-DA domains – they’re relatively easy to manipulate.

    You can read more about DA here: https://moz.com/learn/seo/domain-authority

    Picking a Scraper:

    There are literally hundreds of different domain scrapers out there, and they all have different pros and cons. I'm a believer in testing and, as such, I've tried most of the tools out there.

    A lot of it comes down to personal preference. Some scrapers are billed monthly, others offer one-off payments. Some are web-based, others run on your system. etc etc. I've found the following three to work well in the past.​
    The Scraping Process:

    Like I said above, I'm going to be using PBNLab to guide you through the actual scraping process. This is by far the easiest part of this whole process and can easily be copied across to whatever tool you're using.

    Heck, you could even do this manually if you want.

    [​IMG]
    Find your way over to the "Search by Keyword" option. Give your job a name, and then make sure the country is correct (This is the Google Engine that the scraper will use. Say you want to find a domain for an English shoe shop, it’s a good idea to use Google.co.uk)

    Enter the Keyword you want to search for (Remember what I said earlier) and then press next. After a bit of waiting it shows you the results that it's going to be checking from, and then gets scraping.

    That's it. How simple was that?!

    Now all you have to do is wait for whatever tool you're using to finish checking all of the URLs, and then you'll have a nice list of expired domains to check through....

    Checking Process:

    [​IMG]

    This is where the majority of people searching for domains fail. It takes a bit of dedication to separate the wheat from the chaff, but boy is it worth it! Spending an extra ten minutes on this section can literally be the difference between finding the best domain of your life and another wasted project.

    [​IMG]
    Around 90% of the domains you find won't even be worth checking. I'm not sure what caliber domains you're looking for, but I can usually find a use for domains with 10+ metrics.

    PBNLab gives you the option to filter by TLD, Keyword, RD whatever. There are a ton of options that can help you weed out the worst domains without even moving a finger.

    (It's worth noting that the index isn't always up-to-date, so some domains can easily fall under your filter when it comes to actually checking them)

    History - Archives: One of the quickest ways to tell if a domain's going to be suitable or not is to see what it's been used as in the past. PBNlab makes this really easy and, from the Action drop-down on the right-hand side, you can quickly get to the web archive for it.

    [​IMG]
    Take the domain name "regenwm.org" for example. A quick look at the archive I can see that it's been used as an Asian site, which isn't going to be very useful for a plane blog...

    [​IMG]
    Don't just check one screenshot though. Just because a domain was used for a legitimate site at one point in time doesn't mean it's not been spammed to hell.

    This is by far the easiest step and doing it first will save you a lot of time! Archive.org isn't always available, so just note down the domains that you can't access and come back later.

    Anchors: After making sure the domain was a legitimate site in the past, you need to know what SEO's been done to it. Checking the backlink anchors with sites like Majestics and SEMRush is an easy way to see what sort of KW's have been targetted.

    [​IMG]
    From the above anchor tags, I can tell that this domain's been used for an illegitimate project in the past and there's a greater chance that it's been penalized by Google. One of the easiest ways to check if your site has been hit by Google is to use a free checker ismywebsitepenalized.com or by simply entering "site:domain.tld" (No quotation marks) into Google.

    An ideal domain will have a range of relevant Anchors, coming from a range of different link types (Images, ********, nofollow etc)

    Backlinks: I like to use a mixture of Moz's Open Site Explorer and SEMrush for checking a domain's backlink profile.

    Moz gives me a quick idea of the sort of backlinks the sites has, as well as a semi-accurate spam score (Gives you an idea, at least) and while I'm there I can check the DA and PA of the domain.

    When using these tools, make sure to check the www and non-www version of the domain. Some adjust for this automatically, but a lot of the time you can be looking at a sub-domain with 0 SEO done to it.

    After making sure that there aren't any BL's that immediately stand out on Moz, I head over to SEMrush. Here you can see an in-depth backlink report with the link type, age, authority, anchor, relevancy etc.

    [​IMG]
    A lot of people forget to block SEMRush's crawlers, meaning you may come across a few PBN's. This is not good news. After purchasing one PBN service, here on BHW, I was able to uncover the seller's whole network just because they didn't block the right crawlers.

    If you can find a PBN link pointing towards your site, your competitor can too.

    I'm not saying to avoid all domains with PBN links, but take care. It's a good idea to disavow any preexisting PBN/Bad links when you register a domain.

    Metrics: Like I mentioned earlier, PBNLab's index isn't always up-to-date, which is why checking the metrics again at the end is imperative. The last thing you want is to list a domain for sale that has a Domain Authority of 22 when really it's only 17...

    [​IMG]
    I like to keep a handy table, like the one above, with all of the domains that make it through the earlier tests. Keeping information in a detailed table like this makes it really easy to plug into a marketplace system to sell them.
    SEO Tools: SEO Tools are expensive. Really fucking expensive.

    If you're a big agency that's earning tens of thousands a month from SEO then a few hundred dollars for a metric tool may not seem like a lot, but when you're working by yourself and only registering a few domains every month, paying for subscriptions can really halt your growth.

    I know for a fact that you're not allowed to sell group buys on the forum, but I don't think there's a rule preventing me from talking about them. I use different group-by services for various services, and they're great. For something like $5 a month I can get access to SEMRush (Has a minimum RRP of $99) and when using a service like groupbuyseotools.com you don't have to worry about someone changing the password.

    Here are the SEO services I currently have thanks to different group buys:​
    • Majestic's
    • Moz
    • Ahrefs
    • Indexification
    • DomComp
    ----------​
    Once again, this was an extract from the book I started writing. I finished the first draft and just felt all of the information was a little... obvious, hence the absence of anything groundbreaking from this thread. I know a lot of you will know what you're doing when it comes to hunting for expired domains, but there are still a lot of people who have no idea.

    I've still got to edit out a lot of crap and cut out unnecessary content but there are three more sections I'm going to release later on: A complete walkthrough of starting a site, A complete walkthrough of SEO, and a complete walkthrough of social media.
    ----------​

    Can you actually find good domains like this?

    Yes. Oh boy.. fuck yessss!

    I've listed two high-class domains for sale on BHW in the past, both found using this exact method.

    Literally, a day's scraping and I found three domains around this level:

    [​IMG]
    [​IMG]

    If you're willing to drop your standards and go for domains around the DA 20 area, then you'd struggle to use all of the domains you find. It's honestly amazing how many good domains you can find after a bit of testing and tweaking with domain scrapers. Like I said at the start, even if you aren't interested in using a domain for your own projects, you can make a lot of money by flipping them.

    [​IMG]
     
    • Thanks Thanks x 19
    Last edited by a moderator: May 30, 2017
  2. Nargil

    Nargil Jr. VIP Jr. VIP

    Joined:
    May 10, 2012
    Messages:
    4,601
    Likes Received:
    2,945
    Location:
    Europe
    Home Page:
    Very informative and I agree with most of the stuff.

    Few things I would like to add:

    - As many people who had or will try to scrape expired domains, the scraping itself is the easy part. Determining what is and what isn't usable is a main problem, which then leads to threads like the one from today with kid doing expired domain giveaway and 95% of them being spammed. Thread is deleted by now.
    - There is much more to that than just checking the archive.org, trust me. I uncover most of the crap by other means. Nameserver history is a must. Domaintools screenshots are must as well and checking historic index, especially with higher end domains simply must be done. Archive.org is not enough.
    - CF is quite useless and even if it weren't, you don't want it high, because it's a spam metric. It's all explained in that article of mine that you have posted there.
    - There is a world of difference between measuring TF on root/URL, same goes for RD.
    - You should not focus on DA at all. If you really need it, just put it to 5 or 3 and even that's kinda useless. It's also explained in the article I wrote that's in the opening post here. Focus on RD and TF if you want. Disregard any DA/PA etc. You can thank me later :)

    And one more thing. I go through 2 million domains a month, obviously, I use shotgun approach, but never once in my life, out of 10mil+ domains that I have ever went through, have I seen RD950 domain that would be available for free registration and actually clean. With the current overscraped market situation, getting clean stuff with RD40+ and TF9 is quite a miracle. If you don;t mind, feel free to send it over to me via PM, I would LOVE to have a look. And you will have my eternal respect if that stuff is clean.

    EDIT: Domain was .SG, so that's perfectly fine of course. I thought we are talking purely about top TLD domains here. :) Also, the one I saw would not pass my personal check. Anchors were too spammy for my taste.
     
    • Thanks Thanks x 3
    Last edited: May 8, 2017
  3. Jayder

    Jayder Jr. VIP Jr. VIP Premium Member

    Joined:
    Dec 8, 2015
    Messages:
    1,044
    Likes Received:
    907
    Location:
    Pays-Bas
    Home Page:
    Great job! [2 words]
     
    • Thanks Thanks x 7
  4. what_hat?

    what_hat? Jr. VIP Jr. VIP

    Joined:
    Sep 1, 2012
    Messages:
    215
    Likes Received:
    110
    Gender:
    Male
    Location:
    CET
  5. yolatimez

    yolatimez Newbie

    Joined:
    May 15, 2017
    Messages:
    5
    Likes Received:
    0
    Gender:
    Male
    Great tutorial, helpful to those getting started with domain scraping.
     
  6. Heiko

    Heiko Jr. VIP Jr. VIP

    Joined:
    Jul 26, 2016
    Messages:
    674
    Likes Received:
    439
    Gender:
    Male
    Location:
    Germany
    Exactly what i needed, very informative and straight onto the point.
    Thank you @hasa1015
     
  7. MisterF

    MisterF Jr. VIP Jr. VIP

    Joined:
    Nov 29, 2009
    Messages:
    7,159
    Likes Received:
    5,633
    Occupation:
    Conference Organiser, Business Advisor.,
    Location:
    JADIP
    Home Page:
    Something else for me to sit, read and digest. Thanks for sharing this TUT.
     
  8. alwaystoday

    alwaystoday Junior Member

    Joined:
    Jan 23, 2016
    Messages:
    179
    Likes Received:
    64
    Occupation:
    Self Employed IM and website developer
    You the man hasa thanks for the share
     
  9. Markus Aurelius

    Markus Aurelius Junior Member

    Joined:
    Dec 20, 2016
    Messages:
    189
    Likes Received:
    46
    Gender:
    Male
    Awesome tutorial
     
  10. izkech

    izkech Regular Member

    Joined:
    Jun 7, 2012
    Messages:
    489
    Likes Received:
    153
    Thank you for your article, i picked some precious gems above though i do everythi g manually
     
  11. jiungsong

    jiungsong Jr. VIP Jr. VIP

    Joined:
    Jan 27, 2015
    Messages:
    1,910
    Likes Received:
    951
    Gender:
    Male
    Occupation:
    Living the IM Dream
    Location:
    California
    The word count hooked me in to read this, not bad! [50 Characters]