Think about it like this. You're watching TV, in a second or two you decide that what is on that channel is not worth your time. So, you change the channel to something you like better and you continue to watch. Or, in terms of YT, you change to another video, a video that you find closer to your search term and you keep watching the video either in its entirety or for a fair or longer amount of time. This is how YT determines a 'good' video as compared to a 'bad' video. Being that the on-page optimization for a video is limited (you can only have one page and are limited to keywords, tags etc) it is the largest determining factor in ranking a video. Yes, you can still build backlinks to your video, but if nobody is watching a particular video, it gets lost in the search results. Why try to show something that offers no value to the visitor? So the videos with the higher retention, or length of stay, are seen as more of a benefit to the visitor because most of the visitors stick around to see what the video is offering so therefore they rank better in YT. Call it YTO (YouTube Optimization) (I just coined that, nice!).