Hey everyone, I am trying to make a backlinks datatabase similar to what Ahrefs and Semrush have, but of course on a smaller scale.
The goal is to have something that lets me easily check the backlinks of any domain and other quick metrics like DR, referring domains, total link counts, ******** vs nofollow... without any of the limits(and costs) of the paid tools. I am mainly going to use it to hunt for expired domains, but will probably also later integrate it in a SaaS site.
For now I am trying to reach about 100 billion links as a first public version of the database. It is a low amount compared to some other tools, but, since I prioritize crawling high authority domains first, many of the missing links will be of less importance. I am also thinking of making this first version available to everyone as a free tool, similar to the free ahrefs backlink checker, but where you are able to see all the links for a specific target and sort/filter the results as you want.
Why this isn't easy:
Even "just" 100 billion links is still a huge amount of data. It will require me to crawl billions of unique pages, process their html, extract and normalize the external links, and then store terabytes worth of data, all this with limited hardware. Then there is the much harder task of actually turning all that data into a database that you can query and that is fast. With the resources of an Ahrefs or Semrush, all this would be significantly easier, but with all the hardware restrictions I have I am going to need to get creative.
What I have so far:
I already have stored on my server the backlinks data of a few billion pages that were crawled, and it is already over a terabyte, even with me trying to take as little space as possible. Some of the crawl data comes from my own crawler and some of it comes from publicly available datasets. I already have most of the code that I am going to need done, not just to crawl and process the pages, but to also later turn that data into an actual database.
What is next:
I am going to scale the crawling/processing until I reach about 10 billion pages, which, after filtering spam links, should be close to about 100 billion links. After that, I am going to apply google's Pagerank algorithm to my data so I can get my very own Domain Authority metric !
I am writing the code for that right now: even without having the actual backlinks database ready, once that is done will be able to easily check the authority of all the domains I want.
The goal is to have something that lets me easily check the backlinks of any domain and other quick metrics like DR, referring domains, total link counts, ******** vs nofollow... without any of the limits(and costs) of the paid tools. I am mainly going to use it to hunt for expired domains, but will probably also later integrate it in a SaaS site.
For now I am trying to reach about 100 billion links as a first public version of the database. It is a low amount compared to some other tools, but, since I prioritize crawling high authority domains first, many of the missing links will be of less importance. I am also thinking of making this first version available to everyone as a free tool, similar to the free ahrefs backlink checker, but where you are able to see all the links for a specific target and sort/filter the results as you want.
Why this isn't easy:
Even "just" 100 billion links is still a huge amount of data. It will require me to crawl billions of unique pages, process their html, extract and normalize the external links, and then store terabytes worth of data, all this with limited hardware. Then there is the much harder task of actually turning all that data into a database that you can query and that is fast. With the resources of an Ahrefs or Semrush, all this would be significantly easier, but with all the hardware restrictions I have I am going to need to get creative.
What I have so far:
I already have stored on my server the backlinks data of a few billion pages that were crawled, and it is already over a terabyte, even with me trying to take as little space as possible. Some of the crawl data comes from my own crawler and some of it comes from publicly available datasets. I already have most of the code that I am going to need done, not just to crawl and process the pages, but to also later turn that data into an actual database.
What is next:
I am going to scale the crawling/processing until I reach about 10 billion pages, which, after filtering spam links, should be close to about 100 billion links. After that, I am going to apply google's Pagerank algorithm to my data so I can get my very own Domain Authority metric !
I am writing the code for that right now: even without having the actual backlinks database ready, once that is done will be able to easily check the authority of all the domains I want.
