Two places to speed up a PHP application (which wordpress is) are the server side and the client side.
The server side is controlled by:
1) Number of PHP files included, parsed & executed - so the more options the site has, usually the slower the site. Meaning if the theme is really simple and has one file (index.php, single.php) for serving up pages, then the faster it will be. The more options and choices the scripts are offered the slower it will get. Included .php files have to be pulled off the disc, parsed and executed. So if you have a really fast theme, BUT you have lots of plugins (ie, included .php files to add) activated, then this can surely slow down the site. So no unnecessary plugins, only what is TRULY needed.
2) The number of db queries. How many times the db has to be hit can have a very large impact on the site performance. So if each page has to query the db many times (and it does in wordpress), then the slower the page can be. This can be really exasperated by plugins that do things "improperly". Free plugins (and many premium plugins) come with NO guarantee of good coding practices. So caveat emptor!
The client side:
1) Included external files. Basically CSS & JS files. You have a few places to really slow down or speed up a website. If you search for "Best Practices for Speeding Up Your Web Site", you should get a yahoo page that discusses several issues. I use this as a guideline in building websites. They include using a CDN for standard libraries (like jquery, jqueryui, bootstrap, etc). Combining, minifying & even gzipping your .css & .js files. Matt Cutts (is he considered the devil on this forum?) used to use wp-minify on his site and it did several things to minimize the overall file download size of his site. He no longer uses that plugin, but he does use WP Super Cache. This plugin is great, IF (notice the CAPS) you know how to set it up properly. It basically can cache an entire page as a static .html file and serve that INSTEAD of hitting the database and dynamically creating the page each time (which is wordpress normal). If you do NOT have comments on your site then this is a no brainer. If you DO, then you will need to become intimately familiar with the plugin to configure it properly.
2) Images - do not use a 1024x1024 image when you are displaying it 56x56. A little of an exaggeration, but the point is use the smallest images possible on your site. GIMP (free version of photoshop) is good for reducing image sizes, which can be HUGE when not optimized. You don't want to use images so small they look granular, but you also do not want to embed a 156k image, when 24k is all you need.
So if I was going for performance, here is the logic I would use:
1) Deactivate all unnecessary plugins.
2) Use wp super cache & become well versed with it!
3) If you can customize theme, make all standard js & css libs (jquery, jqueryui, bootstrap, foundation, etc) call out to cdn. Combine all custom js files into one js file and minify it jscompress[dot]com will do it.
4) Ensure images are smallest possible.