Google tracks every click in SERPs, so if a user clicks your link and then press the back button, it means that the exact same set of SERPs will load for that user again, and in Google's eyes, it's a bounce..
My experience is that a bounce rate of more than 80% over a few days, will kill your site in Google..