I haven't noticed anyone mention the actual reason consistency is so heavily stressed when talking about YouTube: your subscribers.
It's not about algorithm. It's not all about growth. Consistency is an aspect of user experience. I'll give you an anecdote: My ex used to watch DIY Youtubers constantly. They all had schedules and would post consistently, on the same day, every week. No matter what. At some point, she had seen all of these youtubers videos, yep, their entire archives. (Isn't this the ideal, to have every sub watch all your videos, maybe even more than once?) How excited do you think she was when a new video came out, on time, as expected? How disappointed do you think she'd be if they didn't?
I watch some podcasters on Youtube who post the same time every day, like morning radio; I like to listen with my coffee and start my day. If they posted at 3am Mon, 5pm next Thurs, 6pm Fri, whenever, however, do you think they would retain more users this way or the former?
Some may not think consistency is important to growth but it is a basic function of human biology as far as your subs behavior: anticipation and expectation. We enjoy familiarity, we enjoy when our expectations are met.
All in all it's about what you're trying to do on Youtube. If you're making one-off How-Tos or Car repairs, I suppose it wouldn't matter much as these are search traffic oriented. Ask yourself, if you were a super fan of your content, what would you prefer?
Edit: To answer your question the number one factor that will contribute to growth is content quality followed by supply/demand. Maybe both equally. You can make really amazing dog foot print paintings but unless your dog is Da Vinci no one is going to watch this, no market. On the other hand, if the content quality was Da Vinci level, then you might have something...