Depends on what type of a site you have. For WordPress hosts such as WP Engine, Kinsta and Seravo offer excellent speeds and you don't have to use those rather poor plugins for caching when it's done correctly on the server side of things.
And of course, bad code, large image files, tons of JS etc. results in slow loading times, so speed optimization is needed. For a site to load in under a second it needs to be served from a HTTP cache, because on the first load you have to have a pretty damn small and static site for it to load in under a second.