Maybe or maybe not. If you are linking out to a bunch of shady sites, you could lose some PR and dilute your sites trust rank which could lower your position. But, it's also entirely possible that you get your PR dropped and it doesn't affect you a bit. This recently happened to Griz (google "make money online" (no quotes); he's the blogspot blog). He got slapped and all his PR taken away but still retained his top 3 ranking.
g00gle seems really fininky about this stuff sometimes. My advice is to go easy on the outbound links if they are to shady spots. But, if they are legit then have at it.