Can this hurt my rankings?

I bought a domain around 3 months ago now which is pretty meaningless to anyone apart from me and someone from uni, it's kind of our names mixed around a bit. Anyway to cut a long story short we were going to build a buy/swap/sell site for our local area. We did some minor SEO on it (bookmarks, a few web2.0s) literally around 100 back links that's all and obviously we never ranked for any major keywords.

Anyway, we decided against that as we just set up a Facebook page which works better. So I am using the domain personally for selling Facebook Likes etc which I am ranking on the first page for most keywords but I have found now I am getting a lot of traffic looking for swap shops and stuff which we don't do now! We are actually ranking 2nd for a major swap keyword and getting tons of traffic from it.

Basically, I know I should of used another domain but I figured because Google reindexed the site as what it is now that the minor SEO we did months ago wouldn't really matter.

Am I wasting my time doing any more SEO on the site or should I stop ranking for the swap keywords eventually?