In my experience, your site is not likely going to drop in rankings due to a "penalty", rather it will drop because it has less link juice going to it. That being said, you don't necessarily need to remove the link right away.
I try not to be too "reactive" to ranking fluctuations. It's never a...