Assembling A PC To Run GPT-Neo/J, StableDiffusion and OpenAI Whisper - Specs?

[noellarkin]

My laptop won't be able to handle any of these, so I'm wondering if it's a good idea to assemble a PC that has the horsepower to do the following:
1. Run one of the smaller GPT3 alternatives (of course its impossible to run the larger ones locally without a TPU)
2. Run stable diffusion for image assets generation
3. Run OpenAI Whisper speech to text
Is anyone running these locally? What system specs are you using and what are the benchmarks?

Also, it seems like (1) would be the hardest and have the highest system requirements. If I decide to run 1) in a cloud, what is an option that doesn't flag me for things like adult/nsfw content? OpenAI keeps clutching it's pearls about nsfw content so I'm not keen on using them.

 

[aurim]

3-4 years old servers are getting so cheap that price/value ratio is great.

I would look into one with a full width PCIe slot so you can add a 12gb Nvidia 3080Ti card.

Only drawback can be fan noise, so if you don't have a spare room for it, just go with regular build and normal/silent cooling.

For the regular build I would go with a MB that can take 2 x higher-end Xeon CPU's. And also the PCIe slot.

 

[noellarkin]

Thanks - - the major factor is the graphics card, I suppose? (and an SSD) - - I remember reading that most of the processing in AI models is done by the GPU, so I'm curious why you've suggested using the high end Xeon CPUs.

 

[Mactronn]

You will need to finetune these models, otherwise are quite useless. If you have a plan to finetune biggest models, 12GB of VRAM will not be enough. With that amount of VRAM, GPU will ran out of memory quite fast. From my experience, for finetuning you will need at least 24 GB of VRAM and 80 GB of RAM.

You can go with GoogleCloud. They charge around $2.0 per active hour. I think they still offer $300 of free credits (CC required).

 

[noellarkin]

biggest models

which models are you referring to when you say 12GB VRAM wouldn't be enough? Bloom? What about more efficient models like GPT-NeoX?

You can go with GoogleCloud.

Thanks, I'll look into this - - they don't have any content restrictions I hope?

 

[noellarkin]

Also, how about these options:
https://vast.ai/https://www.runpod.io/gpu-instance/pricing

 

[Mactronn]

which models are you referring to when you say 12GB VRAM wouldn't be enough? Bloom? What about more efficient models like GPT-NeoX?

Thanks, I'll look into this - - they don't have any content restrictions I hope?

Doesn't matter, the more parameters model has, the more RAM / VRAM you will need. NeoX is a model with 20 billion parameters trained on a pile of 900GB dataset. That's a lot and yes you will need a decent amount of RAM and VRAM--min 50GB of VRAM. With bitsandbytes wrapper, which is an 8-bit optimizer, you can optimize high compute needs, but integration is complex and time consuming.

With 12GB of VRAM, you will able to finetune models with 100-200M parameters such as GTP-Neo with 125M parameters. But do not expect perfect outputs.

If you are new to finetuning, I would suggest you start with small models with approx 100M parameters. You can run GPT-Neo w/ 125M params. on Google Colab for free. Then move to bigger models... The learning curve may be long but it's worth it IMO.

Also, how about these options:
https://vast.ai/https://www.runpod.io/gpu-instance/pricing

It looks like a good deal..

 

[aurim]

I'm curious why you've suggested using the high end Xeon CPUs.

For some reason I thought speech to text will require CPU and I thought you may want to do multitasking.
Seems it runs better on GPU.
Having a CPU reserve is not bad anyways.

However as @Mactronn mentioned I'm afraid working with AI will require vast amount of VRAM which can get expensive.
Cloud solutions are an option, although I have never dealt with it.

I'm running stabledifussion on an 8GB AMD card which was a pain to setup (not being Nvidia), I get one image in about 22 seconds.
Training those models is another story.

I prefer to own things rather than renting, so as GPU prices are dropping I hope I can have my own setup for training too.

 

[noellarkin]

I'm running stabledifussion on an 8GB AMD card which was a pain to setup (not being Nvidia), I get one image in about 22 seconds.

what happens to system performance during the generation, can you use the computer at all or does it freeze up?

 

[aurim]

what happens to system performance during the generation, can you use the computer at all or does it freeze up?

It's just the GPU using up to 90% of power. CPU cores are barely moving so system response is not affected.

 

[Mactronn]

what happens to system performance during the generation, can you use the computer at all or does it freeze up?

It's highly recommended to use linux system. Linux is lightweight, more stable and does not freezes that often. Ubuntu is quite simple to install. 10 minutes and you're ready to go.

In my cases I mostly expirenced CUDA out-of-mem error - you get an error and process of fine-tuning is stopped.

 

[aurim]

You can go with GoogleCloud. They charge around $2.0 per active hour.

Made some more research and gtp neox will take minimum 40gb of ram to train, that's 4 x RTX 3080 10gb that draws 1,3Kw/hour at their peak.
In my country I pay about $0.3/Kw, thus, I will pay around $0.5 per computing hour compared with $2 (also added MB/CPU consumption). Only drawback is investing in those cards.

 

[Mactronn]

Made some more research and gtp neox will take minimum 40gb of ram to train, that's 4 x RTX 3080 10gb that draws 1,3Kw/hour at their peak.
In my country I pay about $0.3/Kw, thus, I will pay around $0.5 per computing hour compared with $2 (also added MB/CPU consumption). Only drawback is investing in those cards.

In theory 40 GB of VRAM, in reality, 50-60 GB.

Additionally, with a cloud-based solution, there is no noise and heat. We started finetuning on a crypto mining rig, and back then the temperature in our office was over 30-celsius degrees. With RTX 3090 or 4090, may be a different story. I haven't tried.

 

[Elvlin]

you need decent gpu with a lot vram. i recommend you buy at least 12gb
If you want 40gb vram or more better buy spesific model that is designed for ai.
i use runpod if i need to train personal model, no problem so far.

 

[Thecodder]

I am interested to buy a pc for AI finetuning and small models training but tought about RTX 4090 Rog Strix OC with i9 13900k and 64 gb ddr5 but unfortunately this GPUs are out of stock in europe
I want to finetune gpt3 alternative models too but i’m afraid even a 4090 is not enough for this task(for a model that return decent content)

 

[noellarkin]

In my country I pay about $0.3/Kw, thus, I will pay around $0.5 per computing hour compared with $2 (also added MB/CPU consumption). Only drawback is investing in those cards.

How does one determine the time taken to fine-tune? Of course it depends on the size of dataset, but is it measured in hours, days, weeks? Also, I'm curious how many epochs to train, and how to decide that number.

 

[noellarkin]

I prefer to own things rather than renting, so as GPU prices are dropping I hope I can have my own setup for training too.

I prefer this as well, and I'm also concerned about things like data privacy when it comes to uploading things like client interaction data to the cloud, if fine-tuning in cloud. Have there been any issues with data breaches and training datasets getting leaked, or data privacy and access issues with cloud fine-tuning? It sounds paranoid, of course, but I have NDAs with one of my clients that makes me a little cautious when it comes to these things.

It's highly recommended to use linux system. Linux is lightweight, more stable and does not freezes that often. Ubuntu is quite simple to install. 10 minutes and you're ready to go.

Thanks, I am a windows user, so I guess I'll just need to have a dedicated system for fine tuning and running these processes.

Perhaps a cost effective solution would involve using cloud based fine tuning and then running it locally.

i use runpod if i need to train personal model, no problem so far.

I f you don't mind sharing, which GPU type did you opt for, and why? Also, did you go for the community cloud or the secure cloud? Thanks.

 

[Elvlin]

I f you don't mind sharing, which GPU type did you opt for, and why? Also, did you go for the community cloud or the secure cloud? Thanks.

depend on use if i want to do intesive training i use 40gb vram. if it's just simple training or just want to generate high res image than 16gb
i always use secure cloud, never try community cloud