Backlinks from a bad neighbourhood won't damage your site.
If that would be the case it would be so easy to wipe out your competition. You only had to put up a spam-site and link to all your competitors, wait till G flags it and everything that's linked to and bingo....