How SEMRush know every keywords for each page ?

Arold79

Newbie
Joined
Feb 28, 2020
Messages
20
Reaction score
2
Hello,
I would like to know how SEMRush know for wich keywords a web page is on Google ?
How do they do that ?
It can't be possible to do this (to much money spending) :
They have a dictionnary of 100 000 words in english (and more in other languages).
They check every request on Google and scrape all the URL.
They match the domain they found on Google with the domain they have in their database.
And do that for each Google.co.uk Google.com Google.de with a different dictionnary per language ?
IMPOSSIBLE ^^ !

How do you think they do please ? :)
Thank you very VERY much :)
 
By writing crawlers and scraping google with bots...
 
Hello PHP-Rockstars ;)
Did you read what I've wrote below ?

t can't be possible to do this (to much money spending) :
They have a dictionnary of 100 000 words in english (and more in other languages).
They check every request on Google and scrape all the URL.
They match the domain they found on Google with the domain they have in their database.
And do that for each Google.co.uk Google.com Google.de with a different dictionnary per language ?
IMPOSSIBLE ^^ !
 
It's possible, they got tons of servers to do that shit. It's not like humans are searching for keywords. Plus data is not always updated. They probably got way more than 100k keywords.
 
They have theor own crawler. May old version of google crawler. Than make algorithm to filter the keyword.
 
Just like my father told me once, work 28 hours a day and everything is possible.

I think they just have 3 or 4 guys, checking every keyword in google, and update the stats.

or maybe they use another keyword service, and just paste the data. easy peasy.

Now seriously, in 2020, how should it be impossible? its all automated so how does it matter how much data is it? you underestimate their budgets to buy servers and software? lol
 
Why do you think these tools are so expensive? Google doesn't have a public API for this afaik so there's no other way to get this data than by brute force.
 
Back
Top