No, but if you leave them then you risk getting kicked from your FB page later on. In addition, you should be aware that there are some tricks as giving a person admin rights and then he/she deactivates the profile - that most of the time results in having a hidden page admin that none of the rest admins can see. So, they can at any time activate their profile and kick you out of the page.
I suggest you do the same, create a ghost profile and then deactivate it and see if it doesn't show in the admin panel. That way you will ensure you always have the control over your page even in case it gets hacked.
Cheers
