There's the metal part - aka technical, what kind of output you're able to get, what options for upgrades and additional services there are - technical performance.
And then there's the "human" part - which starts with sales and contracting, and continues with support and being able to address various requirements - the people behind the hosting company.
If you find two companies that will offer you the similar metal part under comparable conditions, the human part is what will make the difference.
So for me the deal breaker when it comes to reliability and long term value is the people and also the business and contractual conditions. Besides performance and price, I'd look into the support, references, and contractual obligations - as you don't want to get locked into a long term, auto-renewed contract with a lot of small print.
The more established hosts with their own infrastructure, that are able deliver a complex, multi-server infrastructure setup may be able to offer comparable offering and performance, but it's the human and business part that makes the difference..
And contrary to the fact that you can find countless options for hosting if you search in the internet, there's not all that many hosts that would be able to act as a one stop shop for infrastructure, in case your requirements will grow, such as easily scalable servers, flat usage priced CDN, cloud options, now also GPU's etc.
Going back through all the projects I've been a part of since 2005, when it comes to complex server and infrastructure setup, I can first-hand recommend MojoHost.