The term 'fake followers' is too vague to talk about the impact they could have on your account as they come in different categories.
Egg accounts with spammy looking usernames and zero posts.
Half-baked profiles with 10k following and 4 followers.
Fully profiled inactive accounts.
Fully profiled active accounts with some following, followers and posts.
If you have got the latter, you're safe and it will only mess with your engagement rate but if they're really crappy, that could affect your account tremendously in terms of reach as well.