Nowadays most of premium themes are pretty well optimized for the loading time, so I would also care about other important factors too.
Plus, you can purchase the most performing template in the net and then screw it up yourself with poor practices, like calling new files unnecessarily (which implies more requests), uploading videos and images without really optimizing them beforehand, etc.
Installing a cache plugin will definitely boost the speed time of any template you purchase, so I would care more about other points, like user experience (the speed time is one of the most important factors, but not the only one), SEO optimizations (does it respect the basic on-SEO practices? Does it use microdata?), or even if it's updated regularly. You can't imagine how annoying is to get Wordpress updated to the last version and that your template appears to be broken and that you must fix it yourself just because the author is not reachable anymore.