Yes!
Let me explain why.
When your site becomes slow, visitors need to wait a few more seconds than usual. Some of them believe this site is down. Others will go back or close the window to find another alternative site in the search result. So this will affect your site bounce rate because your visitors leave your site before even loading.
Then google algorithm hit your site. Basically, google think if visitors don't like your site then you don't deserve the number 1 spot because your visitors choose other sites over you. That is how site speed affects your ranking.
Actually, I don't think Google really cares about anything other than visitors' needs. If you can make a user-friendly, unique, and useful site for visitors then it will rank even without backlinks.