No, your ranking will not be effected. I completely changed my wordpress theme to something more ad optimized, and my bounce rate decreased over 20%, my site views increased about 50-100 per day, and I'm now making almost twice as much as I was before with the same website.
Now obviously those changes were because people stay on the site longer and click ads more with the new theme. But the point is it definitely had no negative effect on my ranking. I actually think some of my rankings for long-tail keywords increased.