Removing amp needs to be carefully done just so that you don't mess up anything, also you can remove it and reindex using google search console and I don't think it will drastically affect your SEO, your page or site shall not fall extremely low in the SERP's, still, if you feel a bit less satisfied you can further check out some user experiences who have remove amp pages.