There are a lot of things to consider based on your link building, but it all comes down to two things:
1. Try to make things look as natural as possible (where you get links from, diversity in anchors, relevancy - as others have stated)
2. Accept the fact that at one point or another you will get hit - just the way it is