[Ultimate Guide] How to Analyse SERP and Keyword Competition

Holzr

Jr. VIP
Jr. VIP
Joined
Sep 4, 2016
Messages
2,018
Reaction score
2,167
Website
bit.ly
Edit: Some of the pictures seem to be broken for me, can you let me know in the comments if they are for you and I can see about uploading them to somewhere like Imgur and updating the images on this guide.

Does this sound like you:

  1. Find keyword on keyword research tools such as Ahrefs or KWFinder
  2. Filter for low scores/"easy" keywords
  3. Write an article
  4. Wait for the $$$ to roll in
  5. The keyword doesn't rank :(
Unfortunately, this is the approach many people take when doing keyword research, it's what the guides and many gurus say you should do, even the tools say the keyword should be "easy" to rank for.

In this article, I'm going to go over some of the steps I take when evaluating a new keyword I'd like to rank for either in a new niche or one I'm already in. Keyword research is an art form and science and unfortunately, there's no easy way to evaluate the competition. As nice as a score would be, they're just not accurate. You'll see why I'd recommend avoiding these scores and backlink estimates as we go along in this guide...

In order to pick truly low competition keywords, you need to employ a variety of tactics. I'll go over how to analyze SERP and true keyword difficulty both for free and paid. This article is mainly aimed at affiliate terms but pretty much everything can be applied to all sorts of keywords.

This article is packed full of links to some excellent articles that I highly recommend you read if you want to improve your SEO knowledge.

I'll mostly be using Ahrefs to do a lot of the work (it's a great tool) but you can do a lot for free using other tools.

I shall be using the keyword "best running shoes" as my keyword that I shall be analyzing in this guide. It's a high volume search term and by using it I won't be outing any smaller/vulnerable sites, niches or keywords rather than going for a more niche keyword (hopefully that makes sense).

Ahrefs shows a keyword difficulty score (KD) of 24 (Medium) - "you’ll need backlinks from ~27 websites to rank in top 10 for this keyword"

I like to use the following when doing competition research:
  • Backlink/Referring Domain count
  • Allintitle results
  • Relevance/Search Intent
  • Content length/quality and on-page SEO
  • Types of websites ranking
  • SERP History
Tools

Chrome Extensions
  • MozBar
  • Ahrefs Bar
  • SEOQuake
  • SEO Minion
  • Page Load Time
Other Tools
  • Ahrefs
  • Common Sense
I'm going to go in order of easiness for evaluating the competition and step by step (sometimes you won't need to go through them all as you'll figure out the difficulty pretty early on)

Note: I shall be talking about a fair few websites and many affiliate websites. All the ones I am discussing are big media owned websites or have been discussed on the web as case studies etc. I am not trying to out any websites.

Allintitle Results/KGR

A super easy (and free) tactic I like to use to analyze the potential difficulty of a keyword. This is the Keyword Golden Ratio and also the Rule of 63.

Have a read of both articles to see how the formula works.

For results with over 250 monthly searches, you want to use the rule of 63.

Let's calculate the score for our keyword:

allintitle-best-running-shoes-png.115591


277,000 results

Monthly Searches: 52,000

KGR Score: 5.33

You don't even need to calculate the score to know it's not a KGR term. Especially as it's over 250 MS you would need to see 63 results max! Without even looking at any other results it's clear that this is going to be very hard to rank for. You're competing with 277,000 pages for 10 spots.

Now, some people argue against the KGR effectiveness but I believe it carries some weight. I have personally used it as an experiment for my new site and have had a lot of success with it so far (using the rule of 63 for terms over 250 MS). Take it with a pinch of salt, I like to use it as a quick way of evaluating what I'm going up against.

In my opinion, the KGR works best for long tail keywords. Why? Because long tails have a very specific search intent and if there's hardly any allintitle results it's likely that Google is serving the user with an irrelevant result to try and satisfy the search term.

A great example of good search terms is best x for x ones. Often you can find hidden gems that the SERP is not relevant and KGR works really well for these terms. Here's a screenshot for my website, it's got 106k words and is less than 3 months old. No backlinks (I've only just started building this week) - The keywords are all best x for x terms.

best-x-for-x-rankings-png.115592


Note: This applies to both Allintitle and this section. It's important to check the order of words in the keyword and spelling of a term and perhaps the variance of the keyword when evaluating a keyword. E.g. best mens running shoes, best running shoes mens, best mens shoes for running and best running shoes for men. The second result and third result might not show many allintitle results or pages ranking for the exact term. At this point you might think the term is easy, however, it's important to understand the context of the keyword and how Google interprets the search query. Just because the second and third results show low results doesn't mean you can rank for it. In this example results for best running shoes for men will likely dominate the SERP. It's the same meaning as the other terms so you'll be competing with these pages also not just for the obscure order and wording of a keyword such as best mens shoes for running.

Relevance and Search Intent

This is usually the first or second tactic I use. Many years ago when Google was not as smart it couldn't understand, evaluate or interpret what the user's search intent was. Now we have RankBrain (I highly recommend reading this article from Backlinko to understand this section of the article and this tactic) Google has got a lot better of understanding and serving the user's with a more accurate SERP.

Why is RankBrain important? Well, now that SERP results are more accurate anytime you see results that show irrelvant results it's usually a good sign (you'll often find that the result has low allintitle results as well). It shows Google is struggling to show pages that will match the users' search intent.

Edit: Good examples of irrelevant results on affiliate terms are e-commerce stores like Amazon. For best x terms you often see Amazon bestseller pages rank. This is a good sign! The pages are not optimized and just rank on authority.

You'll often see this in best x for x results since these are very specific a lot of articles won't cover them or there be a header in a main best x article. Since H1, Title Tag and URL, etc play a significant part in SEO these terms are often fairly easy to rank for and therefore outrank the less optimized irrelevant results already ranking.

The best x articles that are ranking might be strong in terms of backlinks but this is one of the times you can, to a certain extent, ignore them as you are not directly competing with them (they won't be targeting your exact keyword they're targeting the best x short tail term).

Note: This applies to both Allintitle and this section. It's important to check the order of words in the keyword and spelling of a term and perhaps the variance of the keyword when evaluating a keyword. E.g. best mens running shoes, best running shoes mens, best mens shoes for running and best running shoes for men. The second result and third result might not show many allintitle results or pages ranking for the exact term. At this point you might think the term is easy, however, it's important to understand the context of the keyword and how Google interprets the search query. Just because the second and third results show low results doesn't mean you can rank for it. In this example results for best running shoes for men will likely dominate the SERP. It's the same meaning as the other terms so you'll be competing with these pages also not just for the obscure order and wording of a keyword such as best mens shoes for running.

Looking at Backlinks (Internal and External) and Referring Domains

Take a look at the number of backlinks each of the SERP results has, analyze the backlinks and work out a rough estimate of how many backlinks you'll need to rank. Looking at internal backlinks is also super important and overlooked massively. If you want to learn about the significance of internal links then I'd highly recommend reading the following two articles from Ahrefs.

If you often wonder how some of these authority websites rank in 1 day for a term internal links are your answer. With Ahrefs you can explore the internal links pointing to a page. Many people will see SERP with <10 backlinks and think easy, however, if they have homepage links and you're trying to compete with BIG sites like top ten reviews, best products, etc. think twice.

Back on track, let's look at the SERP for our term:

best-running-shoes-serp-png.115593


So, Ahrefs reckons you'll need 27 backlinks to break into the SERP. Can you see now why you shouldn't take these scores and numbers at face value? Especially if you combine that with the high allintitle results and taking a look at beyond page 1 and beyond many of the results have quite a few backlinks, enough to make you wonder why they aren't in the SERP if Ahrefs is saying you only need 27.

At this point, I'll take a couple of sites at random on the SERP and analyze the backlink quality and relevance.

This article is going to be seriously long if I analyze more than 1 website so I'm just going to take the first result and the third result (running shoes guru)...

The first result (Runner's World) has:
  • 2,381 backlinks
  • 556 referring domains (423 D0follow)
Taking a look at the D0follow links.....

https://www.blackhatworld.com/attachments/runners-world-best-running-shoes-********-rd-png.115608/

For a great guide on evaluating the quality of a backlink check out this article which goes into more detail on the subject than I've covered briefly below.

Those are some seriously strong links. I'm not a huge fan of third-party metrics but it's obvious these sites are strong by looking at both the DR and the fact that most of these sites will be well known to most people. They even have some .edu backlinks.

So, this page has authoritative backlinks, the second important factor to examine is the relevance of those links. Let's take a look at a few of them...

A lot of the domains are general news style domains and therefore aren't 100% relevant but their power means this doesn't matter 100%. A relevant backlink would be from the running shoe manufacturers and running blogs etc. even sports or outdoors websites are semi-relevant.

There's debate over the authority vs relevance of a backlink, however, I believe relevance beats authority. I'm not saying I'd turn down a D0follow link from somewhere like WSJ over a small blog run by a runner but if the running blog was perhaps less authoritative than WSJ then I'd take the running blog link (unless it didn't have any authority of its own and the link wouldn't pass much juice).

Andre Weyher
From what I can see many of the referring domains are pointing at an old page on the Runner's World website which has now been redirected to this new best running shoes article. So, a lot of the links don't make perfect sense in terms of context and relevance but it's clear these links still carry a lot of weight as this page ranks #1.

I'd definitely be put off by this strong backlink profile, it's going to be very difficult to match those number of domains and quality of websites without some seriously incredible content and a big wallet. However, sometimes for these high volume terms, you don't need to rank #1 as most results on the SERP will still get a good level of traffic. In which case for a result like this, you would maybe want to evaluate the backlink profile (and other tactics I will go into below) to see whether there's a chance of ranking perhaps #2-5.

Now, if we take a look at the internal link count for Runner's World Ahrefs shows "132 groups of similar links" including links from the homepage (the homepage is usually where the majority of a websites authority and PageRank/Juice will reside).

So, Runner's World has very strong external and internal backlinks.

Sometimes you might find an article ranking for a term which you think you could improve on and "steal" backlinks from the SERP pages. You could use a host of link building tactics if you choose to go for a keyword in order to gain links to your page. If the page ranking in the SERP looks like it could be improved then you could look at using the Skyscraper Technique in order to "steal" your competitors' backlinks.

Free Method

You can use either MozBar, Ahrefs Bar or SEO Quake in order to get an overlay of the number of referring domains and backlinks a page has.

best-running-shoes-serp-moz-png.115609


It's worth installing all of them and searching for a few results to see which is the most accurate. Since I use the Ahrefs website I can't really comment on how accurate SEO Quake (a SEMrush tool) or MozBar is.

You're limited by free tools in that you won't be able to dive much deeper into the types of links ranking, D0follow vs Nofollow, etc. but using a toolbar is a good start and will give you a rough overview of the difficulty.

To get a rough estimate of the number of links you will need to rank you can calculate the average backlink count per page. If you have MozBar (the others also allow you to do this I believe) and Excel or Google Sheets you can do the following:

Export SERP results

cookie-recipe-serp-snip-export-png.115610

You can then use the average function on Excel or Google Sheets to calculate the average number of links each page has.

Types of Websites Ranking

For this tactic, I'm talking about looking at the types of websites ranking. Sometimes it's very possible to outrank large authority websites even if you have a smaller site with fewer backlinks. In a nutshell, if you create a niche and focused website around a particular topic and cover just about everything (informative and affiliate) you can create an insanely relevant and therefore authoritative website which could outrank much bigger websites. I've personally seen micro niche websites dominate some pretty high search volume terms in the SERP and outrank some very big affiliate and general websites.

Is it a general news style website ranking (Forbes and business insider etc.) - Certainly possible to outrank for some terms

Is it a review website ranking (WireCutter, best reviews, top ten reviews, etc.) - You might have a chance of outranking if these sites are irrelevant to your keyword (perhaps your targeting best wellington boots for people with size 10 feet, I just made up this keyword btw, but all the results are general "best wellington boots")

Are there eCommerce stores ranking? Like I mentioned earlier if you see results for affiliate terms like bestseller pages etc. this is a good sign, they are ranking on authority and are not optimized for the keyword, you should be able to outrank these pages.

Are the sites in SERP thin affiliate websites - if they are and they are fairly niche and only have affiliate content on one topic (e.g. boots) it's certainly possible to outrank these websites

Are the websites authority websites in the niche - for example they are focused on a specific, usually narrow, niche and dominate the SERP with a high concentration of #1-3 spots and cover informative and affiliate content. For example, the website might be dedicated to office chairs and they might have an article on best office chairs and also how to clean a leather office chair. These websites are highgly niche relevant. A little bigger than micro niche websites these websites are trusted by Google because of there narrow focus on one particular niche.

These sorts of websites are going to be hard to outrank (even if their backlink profile doesn't appear to be very strong) and this is the type of website you should be striving to build.

The golden sites to see ranking are social media, Q&A websites, press release websites, web 2.0 (wordpress.com and Blogspot, etc.), article submission websites and forums. These will either be niche specific forums or general forums (e.g. Reddit) or sites like Quora. If you see affiliate terms with these sorts of sites you should jump for joy. They are not optimized and usually don't contain much content, this is a very good sign of low competition.


Content length/quality and on-page SEO

This is fairly straightforward. I like to skim read the page's ranking and use SEO Minion to check on-page factors such as headers, word count, meta and title, are they using appropriate referencing and linking out to authoritative sources etc.

What you're looking for is thin content, poorly written articles with mistakes, poor on-page SEO, slow page load times and an article you wouldn't consider authoritative that you would share or source/cite.

Perhaps it's a best style article but only covers 1 product or doesn't include a buyer's guide or simply misses out basic details and a FAQ section, etc.

Just like the skyscraper technique, think to yourself "could I write content that's better than this"

Sometimes a keyword won't be worth ranking for if the top few results are in incredible depth and there's nothing you can improve. This also goes hand in hand with the tactic above of examining the type of website. Big authority sites and brands will get higher organic CTR because people trust them so will click onto their page, even if it ranks a few positions down. I know I've clicked on a lower position result simply because I know and trust the website more than the other results.

A good example of this would be in the health niche. Say there's an easy keyword about a disease where the SERP doesn't have many backlinks if I saw the NHS ranking I personally wouldn't bother trying to rank the keyword, simply because I know that the NHS article would likely attract a higher CTR even if I outrank it. Note: this is more my personal opinion, you don't have to follow my advice, it's simply just my 2 cents.

Now on the other hand if the SERP is full of affiliate sites (thin style ones, not recognized brands, etc.) and the content is meh then sure I'll give the keyword a go. That's the ideal SERP, one full of thin affiliate websites. People don't have brand loyalty so they'll likely go for #1 result. If I can improve the content then I'll go after the keyword.

Moving onto on-page SEO, you want to see how optimized their pages are for your target keyword.

An important feature of demonstrating E-A-T is using appropriate sources. Google wants to see you cite sources (see p.33 section 6.7). You should do a quick E-A-T evaluation of your competitors pages to see where you could exploit areas they're lacking in. For further reading about the Medic Update and E-A-T take a look at this article.

In terms of on-page SEO factors, you want to look at the URL, Title, Headers, Meta Description and the main content to see if they are mentioning and have optimized for the keyword you'd like to rank for.

SERP History

An underlooked part of keyword research that I rarely hear anybody talk about is SERP history. I don't know of any way of doing this for free. I use Ahrefs keyword explorer for this.

I'm quoting and sourcing from this brilliant article by Ahrefs for this section as I can't really improve on it at all.

"A good way to learn whether Google is happy with the search results or if it’s still figuring out what’s best for users is to look at the SERP history.

Here’s SERP position history that Ahrefs’ Keywords Explorer tool shows for the keyword “search engine optimization”:

11-serp-position-history-seo.jpg


The same 5 pages have been ranking at the top for quite a while, with only small shifts in their positions.

And here’s the SERP ranking history for the keyword “twitter marketing”:

12-SERP-difficulty.jpg


The pages have jumped on and off the Top10 for almost a year. And only recently, Google seems to have figured what kind of search results satisfy people the most.

In other words, SERP position history can be somewhat indicative of Google’s own level of satisfaction with their search results and thus reflect your chances of squeezing your own page onto the Top10."

Using Some Common Sense


Sometimes you come across a term which appears low competition because of the following:

  • A low number of backlinks
  • Small/Thin affiliate websites ranking
  • Overall low domain authority (not the Moz metric but more of a general one from Google's point of view, looking at the overall backlink strength of the website, it's content and how authoritative you deem it to be)
You've used the tactics and methods I've spoken about today and you're confident you've hit the jackpot, the last step in analyzing keyword competition is using some common sense.

Firstly, calculate the monthly value of a keyword. If the keyword is valuable and could easily be bringing in $$$+ a month then you should stop and ask yourself, is this keyword really this easy to rank? Sometimes the answer is yes and sometimes it's now, why? Because often the sites dominating in the SERP will be using PBN links. Go with your gut and have some suspicion if the competition of a keyword seems too good to be true, it's worth taking the time to analyze the SERP in further detail, perhaps by looking for hidden PBN links.

I'm not affiliated in any way with the following tool from @jamie3000 but PBN Hunter is a tool you can use to try and discover hidden PBN links your potential competitors might be using.

Conclusion

I hope you enjoyed this guide and now have some further knowledge on how to evaluate keyword competition and how to analyze the SERP. Have a go at looking for some high search volume affiliate terms and some obscure niches and long tails (with lower search volume, under 1000 monthly searches) to practice.

Examples of high search volume affiliate terms:

  • Best Bluetooth Headphones
  • Best Running Shoes
  • Best Coffee Maker
I hope you enjoyed this guide. BHW was being a little buggy and text was jumping around and paragraphs were relocating themselves everywhere so I really hope I've not missed anything out and that the formatting isn't broken as this guide took a while to write.

TLDR: How to analyze SERP/Competition

  • Backlink/Referring Domain count
  • Allintitle results
  • Relevance
  • Content length and quality
  • Types of websites ranking
  • SERP History
References and Further Reading

 

Attachments

  • allintitle best running shoes.PNG
    allintitle best running shoes.PNG
    10.6 KB · Views: 14,912
  • best x for x rankings.png
    best x for x rankings.png
    126.3 KB · Views: 15,144
  • best running shoes serp.PNG
    best running shoes serp.PNG
    56.8 KB · Views: 14,794
  • runners world best running shoes ******** rd.PNG
    runners world best running shoes ******** rd.PNG
    26.5 KB · Views: 1,616
  • best running shoes serp moz.PNG
    best running shoes serp moz.PNG
    67.3 KB · Views: 14,775
  • Cookie recipe serp snip export.PNG
    Cookie recipe serp snip export.PNG
    183.9 KB · Views: 14,952
Last edited by a moderator:

fasttrak

Jr. VIP
Jr. VIP
Joined
Sep 12, 2016
Messages
2,953
Reaction score
3,313
Website
www.authorityboost.net
Looking forward to going through this properly tonight. Thanks for taking the time to write it all up and share with us.

Edit - I also get broken images (some).
 
Last edited:

jani

Power Member
Joined
Jun 30, 2016
Messages
554
Reaction score
76
nice post to read bookmarked
 

JimHalpert

Supreme Member
Joined
Jan 16, 2016
Messages
1,359
Reaction score
283
Thanks for the share. Going to read in detail.
Thanks!
 
Last edited:

Naijalife

Power Member
Joined
Mar 7, 2019
Messages
526
Reaction score
235
This is really going help me going impliment this tonight bookmarked
 

Stuffedegg

Jr. VIP
Jr. VIP
Joined
Apr 8, 2019
Messages
660
Reaction score
857
Edit: Some of the pictures seem to be broken for me, can you let me know in the comments if they are for you and I can see about uploading them to somewhere like Imgur and updating the images on this guide.

Does this sound like you:

  1. Find keyword on keyword research tools such as Ahrefs or KWFinder
  2. Filter for low scores/"easy" keywords
  3. Write an article
  4. Wait for the $$$ to roll in
  5. The keyword doesn't rank :(
Unfortunately, this is the approach many people take when doing keyword research, it's what the guides and many gurus say you should do, even the tools say the keyword should be "easy" to rank for.

In this article, I'm going to go over some of the steps I take when evaluating a new keyword I'd like to rank for either in a new niche or one I'm already in. Keyword research is an art form and science and unfortunately, there's no easy way to evaluate the competition. As nice as a score would be, they're just not accurate. You'll see why I'd recommend avoiding these scores and backlink estimates as we go along in this guide...

In order to pick truly low competition keywords, you need to employ a variety of tactics. I'll go over how to analyze SERP and true keyword difficulty both for free and paid. This article is mainly aimed at affiliate terms but pretty much everything can be applied to all sorts of keywords.

This article is packed full of links to some excellent articles that I highly recommend you read if you want to improve your SEO knowledge.

I'll mostly be using Ahrefs to do a lot of the work (it's a great tool) but you can do a lot for free using other tools.

I shall be using the keyword "best running shoes" as my keyword that I shall be analyzing in this guide. It's a high volume search term and by using it I won't be outing any smaller/vulnerable sites, niches or keywords rather than going for a more niche keyword (hopefully that makes sense).

Ahrefs shows a keyword difficulty score (KD) of 24 (Medium) - "you’ll need backlinks from ~27 websites to rank in top 10 for this keyword"

I like to use the following when doing competition research:
  • Backlink/Referring Domain count
  • Allintitle results
  • Relevance/Search Intent
  • Content length/quality and on-page SEO
  • Types of websites ranking
  • SERP History
Tools

Chrome Extensions
  • MozBar
  • Ahrefs Bar
  • SEOQuake
  • SEO Minion
  • Page Load Time
Other Tools
  • Ahrefs
  • Common Sense
I'm going to go in order of easiness for evaluating the competition and step by step (sometimes you won't need to go through them all as you'll figure out the difficulty pretty early on)

Note: I shall be talking about a fair few websites and many affiliate websites. All the ones I am discussing are big media owned websites or have been discussed on the web as case studies etc. I am not trying to out any websites.

Allintitle Results/KGR

A super easy (and free) tactic I like to use to analyze the potential difficulty of a keyword. This is the Keyword Golden Ratio and also the Rule of 63.

Have a read of both articles to see how the formula works.

For results with over 250 monthly searches, you want to use the rule of 63.

Let's calculate the score for our keyword:

View attachment 115591

277,000 results

Monthly Searches: 52,000

KGR Score: 5.33

You don't even need to calculate the score to know it's not a KGR term. Especially as it's over 250 MS you would need to see 63 results max! Without even looking at any other results it's clear that this is going to be very hard to rank for. You're competing with 277,000 pages for 10 spots.

Now, some people argue against the KGR effectiveness but I believe it carries some weight. I have personally used it as an experiment for my new site and have had a lot of success with it so far (using the rule of 63 for terms over 250 MS). Take it with a pinch of salt, I like to use it as a quick way of evaluating what I'm going up against.

In my opinion, the KGR works best for long tail keywords. Why? Because long tails have a very specific search intent and if there's hardly any allintitle results it's likely that Google is serving the user with an irrelevant result to try and satisfy the search term.

A great example of good search terms is best x for x ones. Often you can find hidden gems that the SERP is not relevant and KGR works really well for these terms. Here's a screenshot for my website, it's got 106k words and is less than 3 months old. No backlinks (I've only just started building this week) - The keywords are all best x for x terms.

View attachment 115592

Note: This applies to both Allintitle and this section. It's important to check the order of words in the keyword and spelling of a term and perhaps the variance of the keyword when evaluating a keyword. E.g. best mens running shoes, best running shoes mens, best mens shoes for running and best running shoes for men. The second result and third result might not show many allintitle results or pages ranking for the exact term. At this point you might think the term is easy, however, it's important to understand the context of the keyword and how Google interprets the search query. Just because the second and third results show low results doesn't mean you can rank for it. In this example results for best running shoes for men will likely dominate the SERP. It's the same meaning as the other terms so you'll be competing with these pages also not just for the obscure order and wording of a keyword such as best mens shoes for running.

Relevance and Search Intent

This is usually the first or second tactic I use. Many years ago when Google was not as smart it couldn't understand, evaluate or interpret what the user's search intent was. Now we have RankBrain (I highly recommend reading this article from Backlinko to understand this section of the article and this tactic) Google has got a lot better of understanding and serving the user's with a more accurate SERP.

Why is RankBrain important? Well, now that SERP results are more accurate anytime you see results that show irrelvant results it's usually a good sign (you'll often find that the result has low allintitle results as well). It shows Google is struggling to show pages that will match the users' search intent.

Edit: Good examples of irrelevant results on affiliate terms are e-commerce stores like Amazon. For best x terms you often see Amazon bestseller pages rank. This is a good sign! The pages are not optimized and just rank on authority.

You'll often see this in best x for x results since these are very specific a lot of articles won't cover them or there be a header in a main best x article. Since H1, Title Tag and URL, etc play a significant part in SEO these terms are often fairly easy to rank for and therefore outrank the less optimized irrelevant results already ranking.

The best x articles that are ranking might be strong in terms of backlinks but this is one of the times you can, to a certain extent, ignore them as you are not directly competing with them (they won't be targeting your exact keyword they're targeting the best x short tail term).

Note: This applies to both Allintitle and this section. It's important to check the order of words in the keyword and spelling of a term and perhaps the variance of the keyword when evaluating a keyword. E.g. best mens running shoes, best running shoes mens, best mens shoes for running and best running shoes for men. The second result and third result might not show many allintitle results or pages ranking for the exact term. At this point you might think the term is easy, however, it's important to understand the context of the keyword and how Google interprets the search query. Just because the second and third results show low results doesn't mean you can rank for it. In this example results for best running shoes for men will likely dominate the SERP. It's the same meaning as the other terms so you'll be competing with these pages also not just for the obscure order and wording of a keyword such as best mens shoes for running.

Looking at Backlinks (Internal and External) and Referring Domains

Take a look at the number of backlinks each of the SERP results has, analyze the backlinks and work out a rough estimate of how many backlinks you'll need to rank. Looking at internal backlinks is also super important and overlooked massively. If you want to learn about the significance of internal links then I'd highly recommend reading the following two articles from Ahrefs.

If you often wonder how some of these authority websites rank in 1 day for a term internal links are your answer. With Ahrefs you can explore the internal links pointing to a page. Many people will see SERP with <10 backlinks and think easy, however, if they have homepage links and you're trying to compete with BIG sites like top ten reviews, best products, etc. think twice.

Back on track, let's look at the SERP for our term:

View attachment 115593

So, Ahrefs reckons you'll need 27 backlinks to break into the SERP. Can you see now why you shouldn't take these scores and numbers at face value? Especially if you combine that with the high allintitle results and taking a look at beyond page 1 and beyond many of the results have quite a few backlinks, enough to make you wonder why they aren't in the SERP if Ahrefs is saying you only need 27.

At this point, I'll take a couple of sites at random on the SERP and analyze the backlink quality and relevance.

This article is going to be seriously long if I analyze more than 1 website so I'm just going to take the first result and the third result (running shoes guru)...

The first result (Runner's World) has:
  • 2,381 backlinks
  • 556 referring domains (423 D0follow)
Taking a look at the D0follow links.....

View attachment 115608

For a great guide on evaluating the quality of a backlink check out this article which goes into more detail on the subject than I've covered briefly below.

Those are some seriously strong links. I'm not a huge fan of third-party metrics but it's obvious these sites are strong by looking at both the DR and the fact that most of these sites will be well known to most people. They even have some .edu backlinks.

So, this page has authoritative backlinks, the second important factor to examine is the relevance of those links. Let's take a look at a few of them...

A lot of the domains are general news style domains and therefore aren't 100% relevant but their power means this doesn't matter 100%. A relevant backlink would be from the running shoe manufacturers and running blogs etc. even sports or outdoors websites are semi-relevant.

There's debate over the authority vs relevance of a backlink, however, I believe relevance beats authority. I'm not saying I'd turn down a D0follow link from somewhere like WSJ over a small blog run by a runner but if the running blog was perhaps less authoritative than WSJ then I'd take the running blog link (unless it didn't have any authority of its own and the link wouldn't pass much juice).



From what I can see many of the referring domains are pointing at an old page on the Runner's World website which has now been redirected to this new best running shoes article. So, a lot of the links don't make perfect sense in terms of context and relevance but it's clear these links still carry a lot of weight as this page ranks #1.

I'd definitely be put off by this strong backlink profile, it's going to be very difficult to match those number of domains and quality of websites without some seriously incredible content and a big wallet. However, sometimes for these high volume terms, you don't need to rank #1 as most results on the SERP will still get a good level of traffic. In which case for a result like this, you would maybe want to evaluate the backlink profile (and other tactics I will go into below) to see whether there's a chance of ranking perhaps #2-5.

Now, if we take a look at the internal link count for Runner's World Ahrefs shows "132 groups of similar links" including links from the homepage (the homepage is usually where the majority of a websites authority and PageRank/Juice will reside).

So, Runner's World has very strong external and internal backlinks.

Sometimes you might find an article ranking for a term which you think you could improve on and "steal" backlinks from the SERP pages. You could use a host of link building tactics if you choose to go for a keyword in order to gain links to your page. If the page ranking in the SERP looks like it could be improved then you could look at using the Skyscraper Technique in order to "steal" your competitors' backlinks.

Free Method


You can use either MozBar, Ahrefs Bar or SEO Quake in order to get an overlay of the number of referring domains and backlinks a page has.

View attachment 115609

It's worth installing all of them and searching for a few results to see which is the most accurate. Since I use the Ahrefs website I can't really comment on how accurate SEO Quake (a SEMrush tool) or MozBar is.

You're limited by free tools in that you won't be able to dive much deeper into the types of links ranking, D0follow vs Nofollow, etc. but using a toolbar is a good start and will give you a rough overview of the difficulty.

To get a rough estimate of the number of links you will need to rank you can calculate the average backlink count per page. If you have MozBar (the others also allow you to do this I believe) and Excel or Google Sheets you can do the following:

Export SERP results

View attachment 115610
You can then use the average function on Excel or Google Sheets to calculate the average number of links each page has.

Types of Websites Ranking

For this tactic, I'm talking about looking at the types of websites ranking. Sometimes it's very possible to outrank large authority websites even if you have a smaller site with fewer backlinks. In a nutshell, if you create a niche and focused website around a particular topic and cover just about everything (informative and affiliate) you can create an insanely relevant and therefore authoritative website which could outrank much bigger websites. I've personally seen micro niche websites dominate some pretty high search volume terms in the SERP and outrank some very big affiliate and general websites.

Is it a general news style website ranking (Forbes and business insider etc.) - Certainly possible to outrank for some terms

Is it a review website ranking (WireCutter, best reviews, top ten reviews, etc.) - You might have a chance of outranking if these sites are irrelevant to your keyword (perhaps your targeting best wellington boots for people with size 10 feet, I just made up this keyword btw, but all the results are general "best wellington boots")

Are there eCommerce stores ranking? Like I mentioned earlier if you see results for affiliate terms like bestseller pages etc. this is a good sign, they are ranking on authority and are not optimized for the keyword, you should be able to outrank these pages.

Are the sites in SERP thin affiliate websites - if they are and they are fairly niche and only have affiliate content on one topic (e.g. boots) it's certainly possible to outrank these websites

Are the websites authority websites in the niche - for example they are focused on a specific, usually narrow, niche and dominate the SERP with a high concentration of #1-3 spots and cover informative and affiliate content. An example (taken from the Authority Hacker article mentioned above) would be https://pickabow.com/ take a look at the concentration of their SERP positions:

pickabow-ahrefs-traffic.jpg.webp


These sorts of websites are going to be hard to outrank (even if their backlink profile doesn't appear to be very strong) and this is the type of website you should be striving to build.

The golden sites to see ranking are social media, Q&A websites, press release websites, web 2.0 (wordpress.com and Blogspot, etc.), article submission websites and forums. These will either be niche specific forums or general forums (e.g. Reddit) or sites like Quora. If you see affiliate terms with these sorts of sites you should jump for joy. They are not optimized and usually don't contain much content, this is a very good sign of low competition.


Content length/quality and on-page SEO

This is fairly straightforward. I like to skim read the page's ranking and use SEO Minion to check on-page factors such as headers, word count, meta and title, are they using appropriate referencing and linking out to authoritative sources etc.

What you're looking for is thin content, poorly written articles with mistakes, poor on-page SEO, slow page load times and an article you wouldn't consider authoritative that you would share or source/cite.

Perhaps it's a best style article but only covers 1 product or doesn't include a buyer's guide or simply misses out basic details and a FAQ section, etc.

Just like the skyscraper technique, think to yourself "could I write content that's better than this"

Sometimes a keyword won't be worth ranking for if the top few results are in incredible depth and there's nothing you can improve. This also goes hand in hand with the tactic above of examining the type of website. Big authority sites and brands will get higher organic CTR because people trust them so will click onto their page, even if it ranks a few positions down. I know I've clicked on a lower position result simply because I know and trust the website more than the other results.

A good example of this would be in the health niche. Say there's an easy keyword about a disease where the SERP doesn't have many backlinks if I saw the NHS ranking I personally wouldn't bother trying to rank the keyword, simply because I know that the NHS article would likely attract a higher CTR even if I outrank it. Note: this is more my personal opinion, you don't have to follow my advice, it's simply just my 2 cents.

Now on the other hand if the SERP is full of affiliate sites (thin style ones, not recognized brands, etc.) and the content is meh then sure I'll give the keyword a go. That's the ideal SERP, one full of thin affiliate websites. People don't have brand loyalty so they'll likely go for #1 result. If I can improve the content then I'll go after the keyword.

Moving onto on-page SEO, you want to see how optimized their pages are for your target keyword.

An important feature of demonstrating E-A-T is using appropriate sources. Google wants to see you cite sources (see p.33 section 6.7). You should do a quick E-A-T evaluation of your competitors pages to see where you could exploit areas they're lacking in. For further reading about the Medic Update and E-A-T take a look at this article.

In terms of on-page SEO factors, you want to look at the URL, Title, Headers, Meta Description and the main content to see if they are mentioning and have optimized for the keyword you'd like to rank for.

SERP History

An underlooked part of keyword research that I rarely hear anybody talk about is SERP history. I don't know of any way of doing this for free. I use Ahrefs keyword explorer for this.

I'm quoting and sourcing from this brilliant article by Ahrefs for this section as I can't really improve on it at all.

"A good way to learn whether Google is happy with the search results or if it’s still figuring out what’s best for users is to look at the SERP history.

Here’s SERP position history that Ahrefs’ Keywords Explorer tool shows for the keyword “search engine optimization”:

11-serp-position-history-seo.jpg


The same 5 pages have been ranking at the top for quite a while, with only small shifts in their positions.

And here’s the SERP ranking history for the keyword “twitter marketing”:

12-SERP-difficulty.jpg


The pages have jumped on and off the Top10 for almost a year. And only recently, Google seems to have figured what kind of search results satisfy people the most.

In other words, SERP position history can be somewhat indicative of Google’s own level of satisfaction with their search results and thus reflect your chances of squeezing your own page onto the Top10."

Using Some Common Sense


Sometimes you come across a term which appears low competition because of the following:

  • A low number of backlinks
  • Small/Thin affiliate websites ranking
  • Overall low domain authority (not the Moz metric but more of a general one from Google's point of view, looking at the overall backlink strength of the website, it's content and how authoritative you deem it to be)
You've used the tactics and methods I've spoken about today and you're confident you've hit the jackpot, the last step in analyzing keyword competition is using some common sense.

Firstly, calculate the monthly value of a keyword. If the keyword is valuable and could easily be bringing in $$$+ a month then you should stop and ask yourself, is this keyword really this easy to rank? Sometimes the answer is yes and sometimes it's now, why? Because often the sites dominating in the SERP will be using PBN links. Go with your gut and have some suspicion if the competition of a keyword seems too good to be true, it's worth taking the time to analyze the SERP in further detail, perhaps by looking for hidden PBN links.

I'm not affiliated in any way with the following tool from @jamie3000 but PBN Hunter is a tool you can use to try and discover hidden PBN links your potential competitors might be using.

Conclusion

I hope you enjoyed this guide and now have some further knowledge on how to evaluate keyword competition and how to analyze the SERP. Have a go at looking for some high search volume affiliate terms and some obscure niches and long tails (with lower search volume, under 1000 monthly searches) to practice.

Examples of high search volume affiliate terms:

  • Best Bluetooth Headphones
  • Best Running Shoes
  • Best Coffee Maker
I hope you enjoyed this guide. BHW was being a little buggy and text was jumping around and paragraphs were relocating themselves everywhere so I really hope I've not missed anything out and that the formatting isn't broken as this guide took a while to write.

TLDR: How to analyze SERP/Competition

  • Backlink/Referring Domain count
  • Allintitle results
  • Relevance
  • Content length and quality
  • Types of websites ranking
  • SERP History
References and Further Reading


This looks awesome. Looking forward to reading it. Thanks for posting
 

Alex D.

Supreme Member
Joined
Sep 22, 2017
Messages
1,350
Reaction score
720
Excellent guide, you explained everything in details.
Thank you for sharing this.
 

stack paper

Senior Member
Joined
Jan 24, 2008
Messages
899
Reaction score
572
Hands-down the best kw research guide on BHW. It's no surprise he offers the best kw research service on BHW as well ;)

Thanks for all your contributions to the forum, brotha. You da real MVP.
 

Gazmo9

Junior Member
Joined
May 25, 2018
Messages
110
Reaction score
47
Thanks for such a long and well explained guide :)
I have been using some of these points already, but the way you explained makes the process seem even more structured
 

Jarvannis

Regular Member
Joined
Jul 4, 2016
Messages
216
Reaction score
45
Great guide, I will read it again and again when I'll get more time

I hope you enjoyed this guide and now have some further knowledge on how to evaluate keyword competition and how to analyze the SERP. Have a go at looking for some high search volume affiliate terms and some obscure niches and long tails (with lower search volume, under 1000 monthly searches) to practice.

Examples of high search volume affiliate terms:

  • Best Bluetooth Headphones
  • Best Running Shoes
  • Best Coffee Maker

Just to be clear, are you saying these 3 high volume keywords are examples of a good opportunity to try to rank (based on your guide) ? Sorry if it is a dummy question, as I said I will read your guide again with more time, but I ask that just to be sure :p
 

deepdigger

Power Member
Joined
Jul 29, 2017
Messages
688
Reaction score
297
perfect guide but not understand what to do with 250 monthly search volume keyword in to rules of .25 score of KGR you mentioned. its already very low search volume
 

beastkay

Banned - Spamming via PM as well as on the forum
Joined
Sep 20, 2018
Messages
1,185
Reaction score
438
Bookmarked going to read it tomorrow for sure!
 
Top