I don't think it would affect your ranking directly, no. But why not just remove the harsh comments? We know for a fact that engagement (whether positive or negative) is good for the algorithm, so you may be shooting yourself in the foot a bit. Also, usually when I see a video with disabled comments I click off.