Well, let me rephrase it OP. You need to install a linux distribution (I believe Ubuntu can run off a USB key for instance..) on your USB pendrive. This is gonna be your HOST operating system. Therefore when you boot your PC off a USB device you'll see a "linux login" and a full fledged linux box run, right?
Now, since a lot of BH programs won't run on linux you need to install a Virtual Machine, in other words a piece of sw that can run your GUEST operating system (Windows) inside linux. A virtual machine will act as a layer of separation between your physical hardware and the Windows OS whose HD files will be contained in a single file. If you get rid of that file from Linux you'll get rid of the entire emulated Windows. Still with me?
Since Windows will actually run atop of linux if you encrypt the linux file system you'll get an extra layer of separation, therefore an extra layer of security in order not to leave tracks. The cost for all this added security will be in terms of performances that can't be as high as if you ran windows on the real physical hardware.
But if security is what matters the most to you, this is a feasible and inexpensive solution in terms of $$$.