While it is listed officially by Google as a ranking factor, the problem is it is just one of the more than 200 ranking factors.
But I suspect that Google, after the launch of the RankBrain update, is probably paying more close attention to core web vitals like speed.
That's the bottom line.
You have to understand that given a choice between a slow-loading or laggy site and one that loads instantly, users will prefer the faster-loading site even if the slower site has better information or organization or delivers a better experience.
People can't be bothered; that's the reality.
So, yes, speed can be a factor only to the point it impacts user experience.
Focus on user experience.
The good news is there's a lot of workarounds to this.
There's a distribution network.
You can use AMP for mobile.