Had something similar on one of my domains ... well it went only down by a 1000

Most backlinks to that domain were build using directory submissions and PRStorm

So it is clear why PRStorm backlinks don't last but what I have to look into is why the directory links where pulled ...

CPUGeek, how did you build your backlinks?