Manually analysing the top pages in the SERPs is and will always remain the best way,
All of the keywords difficulty metrics just use a small part of the actual SEO factors to give their scores and almost all are based on the page backlinks only,
In other words, they do not check the quality of the backlinks, the domain estimated authority, its internal links & structure, the content used, the number of similar posts/pages and the total of them, and the list go on & on...
If you really want to analyse the competition properly, you'll have to analyse the top 5-15 ranking pages/sites and most of this stuff manually,
Ofc, you'll need some tools like ahrefs to check their link profile, but not the KD difficulty one,
Other than this, you can still use such tools to get an initial idea about the keyword or filter a list of them,
But, it's not enough data to rely on for your niche selection
