It's not so much that it will hurt your blogs but my thinking is, that *real blogs* get comments, so I'm sensible with the comments. At the end of the the day we've all done it, give a little back.
If the comment is sensible and "adds something" I allow it. If it's gibberish or blatant spam, I bin it. Takes me a 30mins in a morning to go through mine saying yes or no. As the vast majority are plain useless gibberish.