Is hosting a large image site on wordpress a lost cause?

I've heard the database get ridiculously slow by the time you reach sizes of over 1000 images and such, and that we're better off with other CMSes or maybe with just plain php. Is this myth or fact?

It depends how big the images are and what plugin you use. If you limit the images to a reasonable size (800 pixels in any dimension), and create multiple galleries, you should be just find. Also if you use a caching plugin like Quick Cache that will also help.