Bounce rate is a user behavior factor of rankings. Although user behavior does not impact search engine rankings as heavy as on-page and especially off-page factors do, it is still counted in Google ranking algorithm. Bounce rate, time on site and other behavior metrics show either your site is sticky or not and are supposed to reflect site's usefulness for its visitors.
However, these metrics alone are not really able to change one's SEPR position dramatically, there are always many other SEO factors involved. That's why uninstalling Google Analytics will unlikely have a dramatic effect, because there are other ways for Google to learn about a site' performance (for instance, Webmaster Tools or visits from SERPs).
More to that, Google Analytics is a really valuable source of data, and if one is not aware of site's performance, traffic sources, conversions and so on - it means some essential information is missing. Of course, there are numerous alternatives to Google Analytics, but it seems there is no reason to complicate things with workaround solutions.