I have built a dozen Joomla sites, and had one hacked. The client whose site was hacked regularly logs on from internet cafes, forgets to lock her FTP on the server and has had her facebook and twitter hacked too. It was fairly easy to stop that happening. The hacker put a small iframe on which appeared to be designed to download a trojan to visitors PCs.
I can pop a Joomla site up in about an hour, including custom template and get a pagespeed of 90/100 on firebug.
I manage a concrete site for a client which I did not build, and there are some basic flaws, such as not being able to set the title meta tag. Well you can set it but it resets 24 hours later to the page title from the database. I'm sure I can fix this when I get round to it, but really, on a default install it doesn't inspire me with confidence. It is otherwise very easy to manage though.
I have only ever set up one wordpress site and was just left feeling that it was a bit of a toy website. This was admittedly a good few years ago, and I'm sure it has improved since then. I have noticed several with a medium/low pagespeed which is gonna hurt google rankings, but I can't comment on if that can be improved as Joomlas can by good quality design.
It'd take something a bit special to lure me away from Joomla now, but it was a bit of a learning curve, and I am good with CSS, and competent with PHP.