Keywords - Ahrefs / SEM Rush - How do they do it?

Hi,

I'm curious about where SEMrush or Ahrefs display the full list of organic keywords for a website. The numbers they report are significantly larger than what Google Search Console provides.

As we're building some features, this is something we haven't been able to figure out yet.

I've attached some screenshots to clarify the data I'm looking for. Would appreciate any insights!

They have bots running checking the SERPs periodically, using different Proxies from different GEOs, different Google TLDs, devices fingerprints, etc...
For the search volume, it's a mix of Google Adwords data and data they buy from ISP providers mainly, just an estimation,
Traffic could be easily estimated based on the search volume & SERP ranking position for the specific page/website

Because these sites have their own scrapers, it's not related to GSC. Their scrapers check all texts on your site and combine them with other competitors keywords. With this strategy they provide so much more keywords than GSC (GSC doesn't combine competitors keywords and it shows yours only).

hideath said:

They have bots running checking the SERPs periodically, using different Proxies from different GEOs, different Google TLDs, devices fingerprints, etc...
For the search volume, it's a mix of Google Adwords data and data they buy from ISP providers mainly, just an estimation,
Traffic could be easily estimated based on the search volume & SERP ranking position for the specific page/website

Click to expand...

Thanks for your help.
Identifying the organic keywords is where we are stuck.
We have been just crawling the Internet (Almost a billion pages a day) for over a year.

amanguy said:

Because these sites have their own scrapers, it's not related to GSC. Their scrapers check all texts on your site and combine them with other competitors keywords. With this strategy they provide so much more keywords than GSC (GSC doesn't combine competitors keywords and it shows yours only).

Click to expand...

Even if it's a new website - that has never been to Ahrefs/SemRush you get this list of keywords even for those.

I can fetch competition for each keyword and marry that to other websites, however the first time when someone searches a website - they are able to show a huge chunk of organic keywords (this has been there since the launch of these software).

Maybe missing something here.

Thanks for your revert.