If you've been building spam links to your site, or your backlink profile consists of too many web 2.0 links, or your content isn't up to scratch then you most likely won't see any difference.
But if you've passed all the tests i mentioned, and you're JUST trying to sort out your site's anchor text diversity with good clean non spam links from non spam sources, on top of other non spammed links that your site already has, on top of good quality lenghty, and well structured articles.. then the chances are that since you've already said that they "have not really made too much of a difference to any of the sites I have tried to rank", that your site's anchor text density was at some stage not as it should have been.. in which case you'll have to wait for a penguin refresh for your rankings to improve since you've most likely already been penalized for having too much exact match anchor text.