Getting crushed by my competition, but I don't know why (Backlinks)

I have a feeling I've completely misunderstood the concept of backlinks.
My competition in google has 4(!) inlinks according to siteexplorer, while I have more than 30.
His backlinks include eHow, Digg, Propeller and one totally unrelated site.

Mine include several high PR social networks and web 2.0 properties, a few high PR forums and some random blogs, mostly PR 2+.

Still, he is on page 1 (#10) while i jump back and forth between page 5 (#1) and page 4 (#10).

Also, this is for my keyword. My site is a .org with the exact keyword in the domainname. He has a different keyword but .com.
His domain is ~2 years old though, while mine is a few months old.
Both have unique content.

So why is it that he ranks so much better than me?
I just don't see what more I can do to rank up.

Just chill.. Google probably isn't finished with you yet, and you're dancing back and forth. Keep getting some backlinks and you'll slowly keep rising until you have the #1 spot.

there are a lot of factors.

1st you said your domain is just months old. that can be it

OR

check your site in webmasters tools if there are any problem with your site and any needs of changing.

Most of the time changing your meta title and description helps a lot. Make it as unique as possible.

older sites generally get a higher preference by google.

Give google some time, and in the mean time keep building backlinks for your site. That ways within some time, you can reach the page 1 spot