brtflixgold
Junior Member
- Feb 13, 2021
- 105
- 22
i want to make website backlink checker like ahref ,semrush or ubersuggest. who know how to make it please tell me
how can i make backlink checker like ahref ,semrush or ubersuggest ?
- Thread starter brtflixgold
- Start date
- Jan 17, 2015
- 5,263
- 4,955
It's very easy. I will give you the simple steps you need to follow:
1. You simply have to create or find a web scraper, that checks the HTML code of any given page.
2. Then extract all the links and save them in a database in the following order for each link found:
Origin URL: the url of the page you have scraped
Destination URl: the Url page that the scraped page links for
3. Then your scraper have to go into each of the URL you have found in the scrapped page.
4. When you go into, you scrape them the same way you did in the step 1
5. And you keep doing this over and over again.
Over a period of some weeks, months, years or decades, depending on your processing power, you will have a ton of registers.
Then you will simply need to do this query:
"Show me all the Origin URL where Destination URL = (whatever you want)"
You only have to consider some little things like not falling in loops, rechecking already crawled pages once and then to see if they have included new backlinks or removed them.
As I said, effortless, It's easy to understand why Ahrefs is so cheap with such easy job done.
1. You simply have to create or find a web scraper, that checks the HTML code of any given page.
2. Then extract all the links and save them in a database in the following order for each link found:
Origin URL: the url of the page you have scraped
Destination URl: the Url page that the scraped page links for
3. Then your scraper have to go into each of the URL you have found in the scrapped page.
4. When you go into, you scrape them the same way you did in the step 1
5. And you keep doing this over and over again.
Over a period of some weeks, months, years or decades, depending on your processing power, you will have a ton of registers.
Then you will simply need to do this query:
"Show me all the Origin URL where Destination URL = (whatever you want)"
You only have to consider some little things like not falling in loops, rechecking already crawled pages once and then to see if they have included new backlinks or removed them.
As I said, effortless, It's easy to understand why Ahrefs is so cheap with such easy job done.
Last edited:
Turbo B.
Elite Member
- May 24, 2015
- 2,854
- 1,636
I think better don't do it.
Question itself gives such answer.
But if you are ready to make it - i can tell how to:
It is very simple - you will need several millions of dollars to rent / buy powerful hardware to scrape the web + professional team of many people to do all job.
And in final you can have site even better than Ahrefs.
Question itself gives such answer.
But if you are ready to make it - i can tell how to:
It is very simple - you will need several millions of dollars to rent / buy powerful hardware to scrape the web + professional team of many people to do all job.
And in final you can have site even better than Ahrefs.
Alexion
Banned - Shitlisted and did not do the right thing
- Sep 9, 2021
- 1,456
- 3,803
Actually, it's quite doable with around $1,000-$1,500 in servers and storage. You don't need it to be comparable to google, as you will be doing a rescan every few days, maybe twice a month even.
At around 600,000,000 urls available, you can scan the database in close to 36 hours, with 5,000urls/s speed. Half a day would be manageble with 6-8 fast dedicated servers. It's the caching of the URLs that would be most problematic. Checking a database when it has tens of millions of rows can be quite tricky.
At around 600,000,000 urls available, you can scan the database in close to 36 hours, with 5,000urls/s speed. Half a day would be manageble with 6-8 fast dedicated servers. It's the caching of the URLs that would be most problematic. Checking a database when it has tens of millions of rows can be quite tricky.
Daniel from Linxact
Senior Member
- Nov 3, 2016
- 852
- 363
I think that most people that answered here, do not have a clue. In theory, building a backlink checker is very easy. As soon as you try it, you will notice that it is very hard. Why? Because of issues that only happen if you deal with large amounts of data.
If you want to achieve a similar result as ahrefs or Majestic, plan to invest at least $500k, if not more, for a first setup and dataset. Ongoing costs will be high. For each dataset creation, you need a few thousand AWS, GCP, DigitalOcean or Vultr instances running for few weeks, each with a different IP address to mitigate IP address blocks and abuse messages. You can imagine how many instances you would need, if you would want to crawl faster.
Just a small list of what you need (there is a lot more I will not mention in this comment):
1. Highly efficient crawling software optimized for every cent in cloud computing costs. Do not even try to use any Java, Python, C# open source crawler. You need a custom C++ crawler optimized for your used infrastructure.
2. Data sanitization pipeline to transform raw data into processed data to store in your NoSQL database like Hbase
3. Apache Hadoop, Spark, Kafka, Hbase (or any equivalent) dedicated root server cluster with at least 0.5 Petabytes and 1-2 Terabytes of RAM without even any historical indexing. Setup costs alone will blow your mind (VLAN, Kerberos, Apache setup, 2FA root login, firewall settings etc.). That amount of RAM is needed to make the data accessible with reasonable throughput and latency.
4. A scalable middleware that is able to handle your users requests and a backend for user data, account creation, login, emails, payment gateway api connection etc.
5. Modern web frontend
6. To even get a chance to make your money back, you need to count in 12 months of ongoing costs without earning a penny and a lot of marketing investments. Your dataset will lose value every day.
Sincerely,
The Data Scientist
If you want to achieve a similar result as ahrefs or Majestic, plan to invest at least $500k, if not more, for a first setup and dataset. Ongoing costs will be high. For each dataset creation, you need a few thousand AWS, GCP, DigitalOcean or Vultr instances running for few weeks, each with a different IP address to mitigate IP address blocks and abuse messages. You can imagine how many instances you would need, if you would want to crawl faster.
Just a small list of what you need (there is a lot more I will not mention in this comment):
1. Highly efficient crawling software optimized for every cent in cloud computing costs. Do not even try to use any Java, Python, C# open source crawler. You need a custom C++ crawler optimized for your used infrastructure.
2. Data sanitization pipeline to transform raw data into processed data to store in your NoSQL database like Hbase
3. Apache Hadoop, Spark, Kafka, Hbase (or any equivalent) dedicated root server cluster with at least 0.5 Petabytes and 1-2 Terabytes of RAM without even any historical indexing. Setup costs alone will blow your mind (VLAN, Kerberos, Apache setup, 2FA root login, firewall settings etc.). That amount of RAM is needed to make the data accessible with reasonable throughput and latency.
4. A scalable middleware that is able to handle your users requests and a backend for user data, account creation, login, emails, payment gateway api connection etc.
5. Modern web frontend
6. To even get a chance to make your money back, you need to count in 12 months of ongoing costs without earning a penny and a lot of marketing investments. Your dataset will lose value every day.
Sincerely,
The Data Scientist
Last edited:
brtflixgold
Junior Member
- Feb 13, 2021
- 105
- 22
Thanks everybody now i know it hard to make for me
- Nov 11, 2018
- 12,084
- 10,839
You can try building this 2 tools i suggested, also there are more suggestions there on that thread:
https://www.blackhatworld.com/seo/l...-tools-any-suggestions.1379012/#post-14941656You don't have to give it for free though , and you can make money.
https://www.blackhatworld.com/seo/l...-tools-any-suggestions.1379012/#post-14941656You don't have to give it for free though , and you can make money.
- Oct 5, 2009
- 2,347
- 1,197
Good Explanation. Finally you made him to take the better decision.
Daniel from Linxact
Senior Member
- Nov 3, 2016
- 852
- 363
Task Eater
Regular Member
- Aug 2, 2018
- 323
- 79
Simply hire a developer.
seomaster5
Elite Member
- Oct 31, 2021
- 5,161
- 1,482
How much is your budget for this ?
Ashk881
Elite Member
- Aug 24, 2021
- 1,695
- 2,071
Pretty sure that they would need a hella lot than 500k to get anywhere near "similar" Results compared to Ahrefs.
Ahrefs has their info about their infrastructure here https://ahrefs.com/big-data
According to that, their cluster has 200k compute cores, about 1.5PB of memory and about 180PB of storage. We talking hugeass data here.
Even taking 10% of the compute resources, a cluster with 20,000 CPU cores and 40TB of memory on AWS costs about $340k per month alone. (1250 instances of a1.xlarge , each costing about 0.4$/hr)
The cheapest I would find on petabyte scale storage was from Backblaze clocking in at $63,000/month for 1PB of storage. Or $630,000 for 10PB of data per month.
Basically about $1 million/month to operate at 10% of the scale of Ahrefs if you rent all your infrastructure.
Should make a pretty decent backlink checker.