For anyone doing prompts offline for whatever reason: guide:
Here's how to push your llm to oblivion.
Run Llama 2 70B on Your GPU with ExLlamaV2
Finding the optimal mixed-precision quantization for your hardware
The largest and best model of the Llama 2 family has 70 billion parameters. One fp16 parameter weighs 2 bytes. Loading Llama 2 70B requires 140 GB of memory (70 billion * 2 bytes).
In a previous article, I showed how you can run a 180-billion-parameter model, Falcon 180B, on 100 GB of CPU RAM thanks to quantization.
Falcon 180B: Can It Run on Your Computer?
Yes, if you have enough CPU RAM
Llama 2 70B is substantially smaller than Falcon 180B.
Can it entirely fit into a single consumer GPU?
This is challenging. A high-end consumer GPU, such as the NVIDIA RTX 3090 or 4090, has 24 GB of VRAM. If we quantize Llama 2 70B to 4-bit precision, we still need 35 GB of memory (70 billion * 0.5 bytes). The model could fit into 2 consumer GPUs.
With GPTQ quantization, we can further reduce the precision to 3-bit without losing much in the performance of the model. A 3-bit parameter weighs 0.375 bytes in memory. Llama 2 70B quantized to 3-bit would still weigh 26.25 GB. It doesn’t fit into one consumer GPU.
Quantize and Fine-tune LLMs with GPTQ Using Transformers and TRL
GPTQ is now much easier to use
We could reduce the precision to 2-bit. It would fit into 24 GB of VRAM but then the performance of the model would also significantly drop according to previous studies on 2-bit quantization.
To avoid losing too much in the performance of the model, we could quantize important layers, or parts, of the model to a higher precision and the less important parts to a lower precision. The model would be quantized with mixed precision.
ExLlamaV2 (MIT license) implements mixed-precision quantization.
In this article, I show how to use ExLlamaV2 to quantize models with mixed precision. More particularly, we will see how to quantize Llama 2 70B to an average precision lower than 3-bit.
Quantization of Llama 2 with Mixed Precision
Requirements
To quantize models with mixed precision and run them, we need to install ExLlamaV2.
Install it from source:
git clone https://github.com/turboderp/exllamav2
cd exllamav2
pip install -r requirements.txt
We aim to run models on consumer GPUs.
Llama 2 70B: We target 24 GB of VRAM. NVIDIA RTX3090/4090 GPUs would work. If you use Google Colab, you cannot run it on the free Google Colab. Only the A100 of Google Colab PRO has enough VRAM.
Llama 2 13B: We target 12 GB of VRAM. Many GPUs with at least 12 GB of VRAM are available. RTX3060/3080/4060/4080 are some of them. It can run on the free Google Colab with the T4 GPU.
How to quantize with mixed precision using ExLlamaV2
The quantization algorithm used by ExLlamaV2 is similar to GPTQ. But instead of choosing one precision type, ExLlamaV2 tries different precision types for each layer while measuring quantization errors. All the tries and associated error rates are saved. Then, given a target precision provided by the user, the ExLlamaV2 algorithm will quantize the model by choosing for each layer’s module the quantization precision that leads, on average, to the target precision with the lowest error rate.
During quantization, ExLlamaV2 outputs all the tries:
Quantization tries for the 10th layer’s up_proj module of Llama 2 13B
-- 1.0:8b 32g s4 8.13 bpw rfn_error:0.00934
-- Time: 19.57 seconds
We can see that the error rate decreases as the quantization precision (bpw, i.e., bits per weight) increases, as expected.
Quantization with ExLlamaV2 is as simple as running the convert.py script:
Note: convert.py is in the root directory of ExLlamaV2
python convert.py \
-i ./Llama-2-13b-hf/ \
-o ./Llama-2-13b-hf/temp/ \
-c test.parquet \
-cf ./Llama-2-13b-hf/3.0bpw/ \
-b 3.0
ExLlamaV2 doesn’t support Hugging Face libraries. It expects the model and the calibration dataset to be stored locally.
The script’s main arguments are the following:
input model (-i): A local directory that contains the model in the “safetensors” format.
dataset used for calibration (-c): We need a dataset for calibrating the quantization. It must be stored locally in the “parquet” format.
output directory (-cf): The local directory in which the quantized model will be saved.
Target precision of the quantization (-b): The model will be quantized with a mixed precision which will be on average the targeted precision. Here, I chose to target a 3-bit precision.
This quantization took 2 hours and 5 minutes. I used Google Colab PRO with the T4 GPU and high CPU RAM. It didn’t consume more than 5 GB of VRAM during the entire process, but there was a peak consumption of 20 GB of CPU RAM.
The T4 is quite slow. The quantization time could be reduced with Google Colab V100 or an RTX GPU. Note: It’s unclear to me how much the GPU is used during quantization. It might be that the CPU speed has more impact on the quantization time than the GPU.
To quantize Llama 2 70B, you can do the same.
What precision should we target so that the quantized Llama 2 70B would fit into 24 GB of VRAM?
Here is the method you can apply to decide on the precision of a model given your hardware.
Let’s say we have 24 GB of VRAM. We should also always expect some memory overhead for inference. So let’s target a quantized model size of 22 GB.
First, we need to convert 22 GB into bits:
22 GB = 2.2e+10 bytes = 1.76e+11 bits (since 1 byte = 8 bits)
We have 1.76e+11 bits (b) available. Llama 2 70B has 7e+10 parameters (p) to be quantized. We target a precision that I denote bpw.
bpw = b/p
bpw = 176 000 000 000 / 70 000 000 000 = 2.51
So we can afford an average precision of 2.51 bits per parameter.
I round it to 2.5 bits.
To quantize Llama 2 70B to an average precision of 2.5 bits, we run:
python convert.py \
-i ./Llama-2-70b-hf/ \
-o ./Llama-2-70b-hf/temp/ \
-c test.parquet \
-cf ./Llama-2-70b-hf/2.5bpw/ \
-b 2.5
This quantization is also feasible on consumer hardware with a 24 GB GPU. It can take up to 15 hours. If you want to use Google Colab for this one, note that you will have to store the original model outside of Google Colab's hard drive since it is too small when using the A100 GPU.
Running Llama 2 70B on Your GPU with ExLlamaV2
ExLlamaV2 provides all you need to run models quantized with mixed precision.
There is a chat.py script that will run the model as a chatbot for interactive use. You can also simply test the model with test_inference.py. This is what we will do to check the model speed and memory consumption.
For testing Llama 2 70B quantized with 2.5 bpw, we run:
python test_inference.py -m ./Llama-2-70b-2.5bpw/ -p "Once upon a time,"
Note: “-p” is the testing prompt.
It should take several minutes (8 minutes on an A100 GPU). ExLlamaV2 uses “torch.compile”. According to PyTorch documentation:
torch.compile makes PyTorch code run faster by JIT-compiling PyTorch code into optimized kernels, all while requiring minimal code changes.
This compilation is time-consuming but cached.
If you run test_inference.py, again it should take only 30 seconds.
The model itself weighs exactly 22.15 GB. During my inference experiments, it occupied exactly 24 GB. It barely fits on our consumer GPU.
Why it doesn’t only consume 22.15 GB?
The model in memory actually occupies 22.15 GB but the inference itself also consumes additional memory. For instance, we have to encode the prompt and store it in memory. Also, if you set a higher max sequence length or do batch decoding, inference will consume more memory.
I used the A100 of Google Colab for this experiment. If you use a GPU with 24 GB, you will likely get a CUDA out-of-memory error during inference, especially if you also use the GPU to run your OS graphical user interface (e.g., Ubuntu Desktop consumes around 1.5 GB of VRAM).
To give you some margin, targeting a lower bpw. 2.4 or even 2.3 would leave several GB of VRAM available for inference.
ExLlamaV2 models are also extremely fast. I observed a generation speed between 15 and 30 tokens/second. To give you a point of comparison, when I benchmarked Llama 2 7B quantized to 4-bit with GPTQ, a model 10 times smaller, I obtained a speed of around 28 tokens/sec using Hugging Face transformers for generation.
GPTQ or bitsandbytes: Which Quantization Method to Use for LLMs - Examples with Llama 2
Large language model quantization for affordable fine-tuning and inference on your computer
kaitchup.substack.com
Conclusion
Quantization to mixed-precision is intuitive. We aggressively lower the precision of the model where it has less impact.
Running huge models such as Llama 2 70B is possible on a single consumer GPU.
Be sure to evaluate your models quantized with different target precisions. While larger models are easier to quantize without much performance loss, there is always a precision under which the quantized model will become worse than models, not quantized, but with fewer parameters, e.g., Llama 2 70B 2-bit could be significantly worse than Llama 2 7B 4-bit
for creating evil demon children or AI gf the benchmarks below will give you a idea.
| Name | Quant method | Bits | Size | Max RAM required | Use case | Compatibility | PC requirements |
| --- | --- | --- | --- | --- | --- | --- | --- |
This probably looks alien to most, but anyone can follow along even on a laptop
We have 3 AI models we want llama2 7b, 30b, 70b, quant really just means quality of communication and tokens that the ai can function under with out spitting out garbage.
The parameters are important it allows the llm to make connections and allow you to SEO, create evil demon children, create what ever you want. While most GPU's are 24gig any higher is more expensive, now this's the train we are all on: use cases for 7b easier and efficient and can be imbedded I'm video games, AI brain, android chat bot, use cases for 30b creating a AI to create code that needs a lot of ram and can act as a gf, the 7b also but you will not get i love you much, use cases for 60+b to create a enterprise business that utilises everything and more and evil kids, 30b and 7b will provide 1 or none, maybe Christian kids.
Example llama-30b.q2_k.gguf is a modified data where it's been compressed and shoved into a box, then another box, and another box losing quality that has been reduced from the default llama2 model saving you a headache and hardware cost as others have done it for you, your free to dl meta llama 2 but that's determined on your PC
Next to that is the quantized new storage to download, followed by vram needed(having 2gig headroom helps)
| llama-30b.Q2_K.gguf | Q2_K | 2 | 13.50 GB | 16.00 GB | smallest, significant quality loss - not recommended for most purposes | low | 32 GB RAM, fast CPU |
| llama-30b.Q3_K_S.gguf | Q3_K_S | 3 | 14.06 GB | 16.56 GB | very small, high quality loss | low | 32 GB RAM, fast CPU |
| llama-30b.Q3_K_M.gguf | Q3_K_M | 3 | 15.76 GB | 18.26 GB | very small, high quality loss | low | 32 GB RAM, fast CPU |
| llama-30b.Q3_K_L.gguf | Q3_K_L | 3 | 17.28 GB | 19.78 GB | small, substantial quality loss | medium | 32 GB RAM, fast CPU |
| llama-30b.Q4_0.gguf | Q4_0 | 4 | 18.36 GB | 20.86 GB | legacy; small, very high quality loss - prefer using Q3_K_M | medium | 32 GB RAM, fast CPU |
| llama-30b.Q4_K_S.gguf | Q4_K_S | 4 | 18.44 GB | 20.94 GB | small, greater quality loss | medium | 32 GB RAM, fast CPU |
| llama-30b.Q4_K_M.gguf | Q4_K_M | 4 | 19.62 GB | 22.12 GB | medium, balanced quality - recommended | high | 32 GB RAM, fast CPU |
| llama-30b.Q5_0.gguf | Q5_0 | 5 | 22.40 GB | 24.90 GB | legacy; medium, balanced quality - prefer using Q4_K_M | high | 32 GB RAM, fast CPU |
| llama-30b.Q5_K_S.gguf | Q5_K_S | 5 | 22.40 GB | 24.90 GB | large, low quality loss - recommended | high | 32 GB RAM, fast CPU |
| llama-30b.Q5_K_M.gguf | Q5_K_M | 5 | 23.05 GB | 25.55 GB | large, very low quality loss - recommended | high | 32 GB RAM, fast CPU |
| llama-30b.Q6_K.gguf | Q6_K | 6 | 26.69 GB | 29.19 GB | very large, extremely low quality loss | low | 64 GB RAM, dual GPU or A100 80GB |
| llama-30b.Q8_0.gguf | Q8_0 | 8 | 34.57 GB | 37.07 GB | very large, extremely low quality loss - not recommended | low | 64 GB RAM, dual GPU or A100 80GB |
Now you're llm cult member.
