Well, they must have ranked and ranked well if you got those views. Did you do keyword research and such or did you get the views some other way?
There has been a HUGE youtube update recently and things have changed a lot. Many people lost rankings, even without affiliate links so this is insufficient data to conclude that posting these links has ruined your rankings, I would say. Give it a bit of time. Rest the channel a bit, revive it later. Track your rankings regularly. Make a new video with and one without an affiliate link. Compare them.
That's how you check things. Empirically.
