I wouldn't worry too much about your spam score, as long as it's not increasing!
Perhaps do a more in-depth analysis of your backlinks, see which backlinks have the highest authority and are relevant to your content and niche and purge/disavow all of the ones you consider to be irrelevant and spammy to your website. Over time, you will see your spam score reducing.