It depends how targeted your visitors are. If you have 10 very targeted visitors a day you could very well have a CTR of 10% for an affiliate link, assuming you have quality content. That's not very much though anyway (one of my sites is like that, link it gets 1 uv a day but it still makes sales every now and then). If you have 100 targeted you can expect a decent ctr for the affiliate product. It's true 100 uqs are not a lot at all, but if they are targeted who cares. You don't want tirekickers anyway, unless you can convert those tirekickers to your list or something.
As for Adsense I have no clue, I got banned a long time ago. That CTR seems very low to me even with only 100 uqs (I know, with adsense its about volume, among other things right?)
With 100uqs a day there's bound to be someone who will accidentally click ads, I know I do all the time on my slower computer, especially when running other programs. Did you try incorporating a new theme if you have wordpress? Some people claimed that with a nicer looking site they got a higher CTR than with adsense traps (you know those really ugly simple adsense sites where people try to disguise adsense ads as a navigational menu and such.). If your bounce rate is low it could be that. That and your content isn't legible or readable (readable as in someone wants to stay and read it because that's what they are looking for).
Hope that helps.