I think your best bet would be to look at what other sites are using. For example, The Pirate Bay and Kickass Torrents are both successful torrent sites. They don't make money from the torrents, simply the ad space. That being said, simply looking at their ads won't tell you which program they use. So maybe a little bit of investigative work will be required there.
The adult niche is sure fire for torrent sites. Just ask the two aforementioned torrent sites. It's been suggested here on BHW time and time again: CrakRevenue. Apparently CrakRevenue is the place to be to monetize adult traffic. I just joined them this morning, I'll report soon on how well they convert, as well as my overall experience.