It is up to you to make the final decision. Are you still making enough money at 4 to be happy? If not, blast it with high PR do follow backlinks. You take the risk that Google may mess with your ranking even more though. If you are making enough money to be happy, just let the website be and don't touch it for a bit longer.