Panda is (arguably) an onsite filter. If you have bad content, an ugly and not user friendly website, high bounce rate, then the site may be hit by this filter.
Penguin is about overoptimized link building. If you build too much of the same anchor text link you may get hit by Penguin. Look at the anchor text distribution of your top ranking competitors and try to copy their numbers.