Yeah it’s possible, especially if your site is still fresh or doesn’t have much authority yet. A sudden spike of toxic or over-optimized links can throw off your profile and trigger algo drops.
Google ignores a lot of junk, but not everything, at scale, it can still cause damage. If it ever happens, keep an eye on your link graph and disavow before it snowballs.