How bad is it?
There are some followers with complete profiles looking real that just stay and don't do any harm.
There are some followers that automatically drop off in a while.
There are some eggs that don't just fade away.
If you're in the first category, manually unfollow a handful every now and then but focus on growing and it shouldn't matter much.
If you're in the second or third category then you might have to get rid of them, still stick to manual unfollowing cause using any app/public bot can be risky for your account.