Had something similar on one of my domains ... well it went only down by a 1000
Most backlinks to that domain were build using directory submissions and PRStorm
So it is clear why PRStorm backlinks don't last but what I have to look into is why the directory links where pulled ...
CPUGeek, how did you build your backlinks?