1. Reliability. Downtime is bad for business.
2. Operating system. Use one that you know. For example, remote administration of Windows usually requires RDP/GUI. Remote administration of Linux can be done over SSH and is usually faster.
3. Supported platforms and languages. Web hosting, in and of itself, is no use to you if it doesn't support the applications you want to run.
4. Bandwidth, tiered pricing, hidden fees, etc. Even large enterprises get bitten by AWS fees, such as its egress fees.