Hello, it is not as serious as it seems, I will give you some possible solutions (or recommendations):
If you use Wordpress and you have a firewall plugin (I use the free version of Wrodfence) it is easy to create a rule to prevent a specific domain from entering your site. Once this rule is created, when a visit to your site from that domain enters you will receive a 503 error, or it could even redirect that traffic to another domain or url.
If you don't use Worpress, you can easily create a rule (either block it or redirect it) for a particular domain in your .htaccess file.
Regarding the drop in your ranking (personally I don't think it's because of this traffic) I would carefully review the profile of inbound links (you can do it from Google Search Console), I would make a disavow file with spam links or bad or seo links Negative, you can submit that file in the same tool and so Google would ignore them ... It's what I would do, I hope it helps you.
(My English is not good at all, bear with me)
