Multiple Pages De-Indexed?

My Wordpress site is 3 months old. It's a travel blog with legit content, and white hat all the way. I write the content myself, take the pictures, and am really putting some effort into creating nice posts. The posts range between 1000 and 5000 words each. I feel they provide value to the reader.

Just 3 weeks ago I had 20 indexed links. I verified this by using site: in Google. I noticed there were an issue where 3 of the indexed pages had an issue. What had happened was the post was published and live, and then later the URLs were changed. Google was showing the page twice with the old URL and with the new URL both. I used the Google tool to "remove" the old URLs and links from the site. A few days later and now I only have 3 indexed links. So I lost about 14-15 legitimate pages somehow. I'm positive I only told Google to remove the old links. There were only a couple.

I've also recently added the Yoast plugin and generated the sitemap and submitted that. It was about 3 weeks ago or longer that I did that. I've reviewed Webmaster tools and I'm showing some 404 errors, but all of those 8 or 9 pages with 404 were pages that were deleted intentionally, etc. and do not need to be in the SERPs.

Can someone tell me what my next step for investigation should be or what the problem is?

If it were me, I would post links on social media sites. Works like a charm to get pages indexed.

Yes. The easiest way is to get social links. There are several decent providers right here on BHW who will give you social signals from aged accounts. That and Lindexed should get most pages indexed. Unless your site is under some kind of a penalty. cheers

Social Signals is your answer, do that and keep up the good work.
Rare to see, people on Travel niche using clicked image.
Would love to check your site, ping me domain name if possible

do some SMO for your site!