For performance, C/C++ is the language of choice at the native level. For the managed world, C#, followed by VB.
However, if you choose to code in the managed world, you will no doubt be met with issues on crossing into the native/COM world, where most of the existing Internet modules for Windows lie.
Don't be fooled thinking that this is just about putting up a WebBrowser control or driving Internet Explorer. It is a hell of a lot more involved than that and in most cases, you are better off spending your time paying someone to build a custom solution, than tackling it by yourself.