Yes, but it's not just the quality that can be harmful. Relevancy, or lack thereof, can also be harmful. Furthermore, you have to look out for context, flow, and other determinants. Sure, links from spam websites are bad, but there are other contributing factors that need to be taken into account.