SSL is a minimum impact on speed, but I think it can also harm your site a little in search engine rankings, because if their is a link that goes to "HTTP" then you have to 301 Redirect it to HTTPS, in which some link juice will be lost.
If you're starting a new site from scratch, then HTTPS is the way to go.