Running Llama 2 locally to create content?

[OneFourNine]

I'm interested in running Llama 2 to create content locally. Does anyone run that here as well?
What would be the specs for 7b, 13b, and 70b?
I'm interested in creating around 10,000 articles per week, which will consume 25 tokens per second for 1 article, one token being 1.33, so the article will be created in 1 minute.
I'm would like to know what are specs that will allow me to do that?
Also, does anyone here runs Llama 2 to create content? If so, what is your experience, the hardware that allows me to do that is the main concern. Some VPS recommandation would be helpfull as well.

I appreciate the help and Happy Holidays. :d

 

[Ashk881]

I have been playing with self hosted LLMs for content creation for some time now. Tho, I don't run any particular model in production as of now because hosting is very expensive (for cloud GPUs as i dont have a GPU of my own.) I have run llama2 on a gpu equivalent to a rtx 3060 (8gb vram) and about 16gb system ram on a intel core i7 machine.

Note that this was with a 4-bit quantized model (i was using the GGUF quantized model), there was some reduction in quality compared to the unquantized original fp16 weights released by Meta but with some RAG I believe you can get some decent quality content.

 

[Fiducia]

Unless you have A40 (2-7b, 2-13b) / A100 (2-70b) GPUs laying around in your backyard, you'll be better of to just rent the GPU time or opt for GPT-4. 10K/articles per week = 47 GPU minutes / day is nothing.

 

[CyberGen]

Unless you have A40 (2-7b, 2-13b) / A100 (2-70b) GPUs laying around in your backyard, you'll be better of to just rent the GPU time or opt for GPT-4. 10K/articles per week = 47 GPU minutes / day is nothing.

How did you calculate that 10K/articles per week = 47 GPU minutes / day ?
10K articles a week = 1429 articles a day.
If one article is in average 1000 words. How can 47 minutes of GPU brings you to generate 1000 * 1429 = 1 429 000 words.

 

[Fiducia]

How did you calculate that 10K/articles per week = 47 GPU minutes / day ?
10K articles a week = 1429 articles a day.
If one article is in average 1000 words. How can 47 minutes of GPU brings you to generate 1000 * 1429 = 1 429 000 words.

1430 articles x 2 seconds/article = 2860 seconds / 60 seconds/minute = 47 minutes on a A100

 

[Larry Igna]

https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF
For most models, you have Model card tab which tells you how much VRAM is required for each file provided. The link above has different quantized llama 2-7b models with specs for each.
Just so you know, I'm able to run TinyLlama on my phone, 4gb RAM. It's not very good for chat, but it's fast for completions. Anyways, having 10-15gb RAM on your PC should get you up and running. Look for models from TheBloke on HF

 

[CyberGen]

1430 articles x 2 seconds/article = 2860 seconds / 60 seconds/minute = 47 minutes on a A100

On A100, it takes 2 seconds to generate 1000 words? This is some 500 tokens/sec speed. I am not sure it is possible by using only an A100 on a llama 2 model, but maybe I am wrong.

 

[Zepph]

It's definetely possible, if you use a low parameter model. I'd recommend Mixtral for this task, it's a Mixture of Experts (MoE) model of about 46B parameters, but uses about the same resources as a 13B model due to its MoE architecture. If you are serious about local LLM, get an RTX 4090 and that should set you up with enough tokens per second. If you're not so sure, instead opt for a cloud provider, such as runpod, vast or tensordock (I use this last one).
If you don't want to mess with setting up the cloud environments and the local APIs to the Mixtral model, then use a provider like openrouter, in this case you'll only need a python code to make it all work together, since the provider takes care of the rest.

Note: I recommend Mixtral due to its amazing performance, lots of people say it's on par with gpt-3.5. If you'd prefer a 7B model, use Mistral 7B and a good fine-tune. In this case, you can't go wrong with OpenHermes 2.5 or Starling-Alpha.