It does depend on how intrusive it is. With me, it always depends how keen I am to get to the content ? if it's a site I'm keen to read, I would usually just click it away and keep on reading. If I have come there from Google and have a few other choices in other tabs, it's the kind of thing that might make me shut the tab and go somewhere else. So it could definitely affect your bounce rate.
As you say, you've got the #1 result on your two main keywords, so I'm assuming it's got content people want to read. So as long as the box is fairly unintrusive, it may at least be worth experimenting with.