Is my website penalized?

I'm working on my website and got many spammy backlinks. The rankings are falling for that particular page. However, I transferred my website from http to https. The results for that page are that, "http" version disappeared from google search and google webmaster on 16 June and the "https" page started appearing instead of previous page. The backlinks for http version also disappeared from search console.
However the rankings are greatly reduced and still going down.
Should I work on the http page or not? Is it possible to recover ranking making more backlinks or I have to work on new domain? My website is 2 years old.

It's not penalized, it's due to you switching to https.

There's been a few threads about this on here this past few months.