VPS. I just started using one a few weeks ago... Really wish I had dropped the money for one sooner! I went with Windows Server 2012 R2 and got a deal at 90 bucks for the year... Now I always have my remote desktop open on one screen and my regular machine open on the other. Totally worth the money and now my home IPs arent always getting hammered.
Most VPS are indeed virtualized machine on a dedicated server. However, VM can also be created on your computer, thus that would be a little different than a datacenter-hosted VPS.
Of course, datacenter-hosted VPS would run 24/7, but make sure you pick a reliable provider, as the smaller ones tend to overload their machines or perhaps use old server hardware that would not run very quickly when operated. A VPS with quality SSD can outperform most old machines, even though the old machine uses 16-core processor and conventional rotational drive.
A self created VM would most likely be on your own computer - if you shutdown your computer, that gets shut off as well.
If you want a machine that runs 24/7, the datacenter-hosted VPS would be a good choice - provided it is from a good provider that is.