The niche is small but the website is complex in regards to keyword research, there are hundred of keywords that would match, the website has almost only user generated content so keyword optimizing isn't really an option.
unless you're big like Reddit or LinkedIn I don't think a UGC site will cut it due to the lack of content first and foremost, but also due to lack of optimized content as well...
And what do you mean by "niche is small"? Small as in there's very little traffic in it, or what exactly do you mean by small?
Also, how many results does google show when you type
site:mywebsite.com into google (obviously replace "mywebsite.com" with your actual website). Because if only 4 results show up it's usually an indication of a penalty and this explains why you're not ranking...
What I read during all the years but never put into action, I should have 5 - 10 wordpress sites, each optimized for one of the biggest keywords and link from these sites to the money site. But it is a lot of work and the costs would be roughly 5 - 10K for each website with linkbuilding and seo text writing. That's a lot of money.
what you're talking about is PBN (Private Blog Network) and, while PBNs work wonders, the problem - like you said - is that it costs a lot of money and dedication to create and keep them safe, because one of the priorities for google's spam-fighting department is to detect and penalize PBNs. So you're going to need to be very careful with your PBN.
Anyway, if you want to go this route I can link you with a guide to setting up PBNs properly that's been shared on the forum, but again, setting up and maintaining a PBN is very cost and time intensive and it's only worth the trouble for big competitive niches. So I don't recommend that you go this route, but if you absolutely must, let me know and I'll link the guide...