How big is your site? Whats the traffic you have at the moment? Unless its really really huge, you dont really need such a big ass server. And Apache and mysql out of box is kind of optimized enough not to slow down your website. Any modification will only shave milliseconds. What you must be doing is to optimize the script that you run. I mean the CMS, wordpress or joomla or whatever you are using.
Let me know. You can ping me in my skype too. My id is repulsorbhw
Cheers!