Hate to say this but karma is subjective.
When something bad happens to an enemy or someone you dislike, we will rejoice and say "HAHA KARMA!" Especially when we know they did something bad before 'karma' ultimate got them.
But when something bad happens to someone you like or yourself, suddenly it isn't about karma anymore. It is suddenly about "bad luck" and "God is just putting you on a test... stay strong."
I can go on about probabilities/odds to prove that karma is just an illusional comfort for victims against their enemies, but I won't go to that.
What I can say is... if someone wrongs you, gives you a hard time, or malign you just smile and ignore them. People do all these either because life hasn't been fair to them and they want to vent it on others; or, in the past, you've committed something and your reputation is stained, so whatever you do from then on you get scrutinised. For example, an ex-convict is your roommate or is a colleague at work, and one day, something goes missing. By human's scepticism and nature, we will automatically assume it is the ex-convict's doing. The prejudice and scrutiny placed on them. I'm not saying we will do it, but others will - it is human nature. Anyway, for your case, when the tables are turned or when 'bad luck' strikes them, it would be your turn to get back on them or just move on. But always remember people have their own reason to do so (like I said, either their life sucks and they want to vent it on others to feel better about themselves, .etc) and no matter what, people will always have something to say so just don't bother and focus on your life. I know it is hard but it takes some time to master this. Transmute your hate and dissatisfaction into something more productive, so you can use this to prove them wrong one day (if you want).
Good luck and stay happy!