Quantizing a Local Model Without Guesswork: How to Choose 4, 6, or 8 Bits Based on VRAM, Context, and Acceptable Loss

The most common mistake when choosing a local model is comparing the size of the quantized file with the GPU memory and considering the decision resolved. That calculation only covers, approximately, the weights. During inference, the key and value cache —KV cache—, activations and temporary buffers, space reserved by the runtime, the context of each request, and, in a service, simultaneous requests also intervene. A configuration might load the model, answer a short question, and still fail when it receives a long document or multiple requests at once.

The practical consequence is important: "fits" must mean that it completes the maximum expected load with a measured margin, not that it starts an isolated session. If the margin disappears, the result can be a memory error, an automatic reduction of the context, offloading of part of the work to system RAM, or very irregular latency. Which of those behaviors occurs depends on the runtime and its configuration; it should not be assumed without checking.

Quantization is one of several levers. Reducing the bits of the weights usually frees up memory and may allow using a larger model, but it does not by itself eliminate the growing cost of the KV cache when expanding context or concurrency. It can also modify quality, performance, or the available execution paths depending on the format and backend. That is why there is no universal equivalence between "4 bits", "6 bits", and "8 bits".

This guide starts from a specific workload: what input length must be accepted, how many tokens are generated, how many requests will coexist, what latency is useful, and what errors would be unacceptable. If you are still choosing a model family, it is advisable to first review the local models guide and, if the doubt is between different base models, use the comparison section before attributing to quantization differences that come from the model.

Do not adopt a quantization just because the model loads. Adopt it only if it maintains the target context, concurrency, and quality under a repeatable load, with enough free memory to absorb runtime variation.

Quantizing is representing real values with a small and discrete set of numbers to use fewer bits and operate with them faster, at the cost of introducing a rounding error [1]. In 8-bit linear quantization, ONNX Runtime documentation expresses it as value_fp32 = scale × (quantized_value − zero_point): the scale is a positive real number and the zero point is the quantized value that represents exactly zero. In the symmetric variant, the scale is calculated with the largest absolute value; in the asymmetric one, with the range between minimum and maximum [3]. The work by Wu et al. from NVIDIA reviews how to choose those parameters and proposes an 8-bit flow that maintained accuracy to less than 1% of the floating-point reference in all the networks they studied [2].

In an LLM, one does not quantize a single thing, but three, and each behaves differently. Weights are fixed, can be converted once, and occupy most of the memory when loading. Activations change with each input and are harder to quantize because they contain very large outlier values. The KV cache grows with context and concurrency. The WxAy notation summarizes the combination: W4A16 means 4-bit weights with 16-bit activations, and W8A8 means 8-bit weights and activations; vLLM, for example, documents FP8 W8A8, INT8 W8A8, and INT4 W4A16 as distinct schemes [27].

Quantizing only the weights reduces memory and, since token-by-token generation is limited by memory reading and not by calculation, it can also reduce latency. GPTQ and AWQ document this: the former attributes its accelerations to less data movement, not to operating with integers [5], and the latter points out that, in FP16, generation has an arithmetic intensity of approximately 1 operation per byte read [6]. Quantizing activations as well allows using integer or FP8 calculation units, but forces dealing with outlier values: in SmoothQuant, some activations are about 100 times larger than most values [7], and in LLM.int8() canceling the few outlier dimensions worsens perplexity between 600% and 1000% [4].

Granularity matters as much as the number of bits. A single scale factor per tensor is cheap, but a single extreme value ruins the resolution of all others. That is why practical formats use a scale per channel or per group of weights, for example every 128 [5][6]. Dettmers and Zettlemoyer found that reducing the block size, that is, dividing parameters into blocks that are quantized separately, is one of the few clear improvements in the balance between bits and accuracy [9]. The consequence is that "4 bits" are never exactly 4 bits: Q4_K stores 4.5 bits per weight and Q6_K, 6.5625 [22], and the Q4_K_M measured by llama.cpp in Llama 3.1 8B reaches 4.8944 bits per weight [21].

A useful estimate starts by decomposing memory into items that can be observed separately. The first is the model weights. Their size depends on the number of parameters, the quantized representation, and format-specific metadata, such as scales, blocks, or auxiliary structures. Therefore, simply dividing parameters by eight, six, or four offers an orientation, but does not replace the actual size reported by the format and the runtime.

The second item is the KV cache. In an autoregressive decoder, the system retains keys and values of already processed tokens to avoid recalculating them for each generated token. Transformers documentation describes cache tensors with dimensions of batch, heads, sequence length, and head dimension. There are keys and values, and this storage is repeated per layer. At the same architecture, growing the context, batch, or concurrency increases the necessary memory.

The third item gathers activations and temporary buffers. Their size depends on the backend, calculation precision, kernels, prefill of long inputs, generation length, and how requests are grouped. It is not advisable to substitute it with a universal constant. The fourth component is memory not directly attributed to the model, such as execution context, libraries, allocators, and fragmentation. The fifth is an explicit operational margin: memory that is deliberately not assigned to tolerate peaks, differences between measurements, and real load.

In a server, batch and concurrency deserve additional precision. A batch can be the number of sequences processed together in a step, while concurrency is the number of live requests. Depending on the scheduler, both figures can be related but are not interchangeable. To estimate the cache, one must count the sum of live tokens of the coexisting sequences, not just the maximum length of a request.

Items that must be recorded before deciding

The estimate does not aim to predict every byte: it aims to discard unviable configurations and define which tests deserve to be run. For weights, use the size reported for the specific artifact to be loaded, not a generic figure for the model family. If only the number of parameters is known, treat the result as an incomplete theoretical minimum. Quantization formats store additional information and some runtimes convert or duplicate structures during loading.

For a Llama-like architecture, a conceptual approximation of the KV cache per sequence is: layers multiplied by two, multiplied by key-value heads, multiplied by sequence length, multiplied by head dimension and by bytes of each cache element. The factor two represents K and V. For multiple simultaneous sequences, the resident tokens of each are summed. If the runtime uses a static cache, it may reserve capacity up to a maximum even when instantaneous usage is lower; if it uses a dynamic cache, usage can grow with the request. Both strategies require measurement.

It is crucial to use the number of key-value heads, not always the total number of attention heads. In multi-query or grouped-query attention, several query heads share KV projections. This can substantially reduce the cache compared to an architecture with one KV projection per attention head. The exact model configuration must provide layers, KV heads, and head dimension; they should not be deduced from a commercial name.

The bytes per cache element must also be verified. Quantizing weights does not automatically imply quantizing the KV cache. Some environments allow choosing a quantized cache data type, others use a different precision by default, and others apply offloading. These options change the memory budget and can affect performance or numerical behavior. Note the actual runtime configuration, not just the bits that appear in the file name.

Conceptual example of cache calculation

Suppose a model with 32 layers, 8 KV heads, head dimension 128, and cache of 2 bytes per element. For a sequence of 8,192 tokens: 32 × 2 × 8 × 8,192 × 128 × 2 bytes, approximately 1 GiB of KV cache. Two simultaneous sequences with that length would approximately double that item. It is an estimate of the cache, not of all VRAM: weights, temporaries, runtime memory, and margin are missing. If the cache is stored with another data type or pre-allocated, the effective result changes.

The previous formula gives the KV cache with 2 bytes per element. To see the order of magnitude with real architectures, the following table applies the same formula to the parameters published in the configuration files of each model [30]. The difference between Llama 2 7B (32 KV heads) and Llama 3 8B (8 KV heads) is a factor of 4 per token, which corresponds to grouped-query attention (GQA): several query heads share each key and value head [20].

KV cache per sequence in 16 bits, calculated with the formula of this article

A detail of the table matters for sizing: with 131,072 tokens of context, the cache of a single Llama 3.1 8B sequence occupies more than the weights of the same model in Q4_K_M (4.58 GiB), so lowering the bits of the weights does not solve long context. In Llama 3.1 70B, 40 GiB of cache per sequence are equivalent to a bit more than half of an 80 GB GPU.

Quantizing the cache attacks that term. KIVI proposes an asymmetric 2-bit quantization without fine-tuning: keys are quantized per channel and values per token, because their distributions are different. Its authors report a maximum memory usage 2.6 times smaller, including weights, batches up to 4 times larger, and a performance of between 2.35 and 3.47 times in real loads [17]. KVQuant combines per-channel quantization of keys, quantization before rotary position, and outliers isolated per vector; with 3 bits it reports a perplexity degradation lower than 0.1 in Wikitext-2 and C4, and that LLaMA-7B can serve a context of 1 million tokens on a single 80 GB A100 [18]. These are results from the authors, with their own kernels and models: they serve to know what is possible, not to promise what you will get in your runtime.

The tools already expose this. Transformers includes a quantized cache with hqq (1 to 8 bits) and quanto (2 and 4 bits) backends, and warns that it can worsen latency when the context is short and the GPU has plenty of memory [25]. TensorRT-LLM allows an FP8 KV cache and, with prior quantization, NVFP4 [28], and vLLM lists quantized KV cache among its options [27]. If your problem is cache fragmentation and waste, and not the size of each element, the approach is different: PagedAttention manages the KV cache in blocks to better utilize memory in service [19]. In any case, the article's rule remains: check separately whether quantization affects weights, KV cache, or both, and measure quality with long context.

There are two families. Post-training quantization (PTQ) converts an already trained model, with little or no retraining and, sometimes, with a small calibration set. Quantization-aware training (QAT) retrains or distills the model simulating quantization. ONNX Runtime recommends resorting to QAT when no PTQ reaches the desired accuracy [3]. For LLM, LLM-QAT observes that PTQ methods work well up to 8 bits and break with fewer, and proposes a data-free distillation with generations from the model itself [31]. For those who run third-party models locally, the practical decision is almost always to choose between artifacts already quantized with PTQ.

The baseline is rounding to the nearest value (RTN). Other methods attempt to reduce the error that RTN commits on the few weights or activations that matter most. The table summarizes the most cited ones and what verifiable data their source provides.

LLM quantization methods and what each source documents

Three lessons repeat in these works. The first is that weights are the easy part and outlier activations, the difficult one: LLM.int8() and SmoothQuant start from that same problem and solve it in opposite ways, isolating outliers or distributing their magnitude with weights [4][7]. The second is that protecting a tiny fraction of channels is enough to recover most of the 4-bit quality [6]. The third is that accuracy improvement usually costs complexity: the fewer bits used, the more the method needs calibration, specialized kernels, or both, and that is why the format must be compatible with your engine, as detailed later.

As a general rule, reducing the precision of the weights reduces their memory footprint compared to a higher-precision representation of the same model. This can make a smaller GPU viable, leave more budget for context, or allow more concurrent requests. However, the observed reduction does not have to follow an exact proportion of eight to six to four. Packing, group scales, file format, internal conversions, and backend buffers change the result.

Quality does not depend solely on the number of bits either. The quantization algorithm, group size, which tensors receive special treatment, the base model, and the task matter. A 4-bit quantization of one method may preserve a task well, and another 6-bit one of a different method may not; the reverse is also possible. For that reason, bits serve to formulate test hypotheses, not to certify accuracy.

Speed requires the same caution. Less memory can reduce transfers and improve viability on a limited device, but a format may lack efficient kernels in a specific backend or require conversions. Increasing the context can shift the bottleneck to cache management and prefill. Performance must be measured with two separate metrics: time to first token for representative inputs and subsequent generation rate. A single figure of tokens per second hides relevant differences.

As a starting point, try 8 bits when critical quality and budget allow; 6 bits when it is necessary to recover a significant part of the memory without jumping directly to the most aggressive option; and 4 bits when VRAM is the dominant constraint or when tests demonstrate that there is no unacceptable loss. These are testing priorities, not universal recommendations.

The following figures are published measurements, not estimates from this article. They start with size: the table reproduces llama.cpp documentation for Llama 3.1 8B. The generation column comes from the same README, which does not indicate the hardware of those tests, so it only serves to compare rows against each other.

Llama 3.1 8B in GGUF: size and speed measured by llama.cpp [21]

Two observations. The first is that Q8_0 occupies a bit more than half of F16 and Q4_K_M, less than a third: the memory savings follow the bits per weight. The second is that speed does not follow the bits: IQ1_S and Q2_K generate almost as fast (79.73 and 79.85 tokens/s) despite Q2_K using 1.2 more bits per weight, and Q3_K_M generates at a speed almost identical to Q4_K_M (71.68 and 71.93). Nevertheless, all quantized versions clearly outperform F16, which generates 29.17 tokens/s in that same test. The study by Jin et al. summarizes it thus: quantization saves memory, but it can also slow down inference, and taking advantage of it requires engineering work and hardware support [12].

A second set of measurements, made with Transformers on a model of about 15.5 billion parameters, shows the actual maximum memory against the rule of two bytes per parameter.

Maximum GPU memory of bigcode/octocoder in Transformers [26]

The Hugging Face guide containing that table exposes a useful rule: the weights of a model of X billion parameters occupy about 2 × X GB in bfloat16 and about 4 × X GB in float32 [26]. By arithmetic, with 8 bits the factor drops to 1 × X and with 4 bits to 0.5 × X, before adding scales, KV cache, and runtime memory.

Now, the quality loss. The table gathers four studies with different metrics, so their rows are not comparable to each other: each row compares the method with its own 16-bit reference.

Quality loss published by the authors of each method

More recent comparative studies look at models and tasks closer to real use. Kurtic et al., with more than 500,000 evaluations over the entire Llama 3.1 family, conclude that FP8 (W8A8) is practically lossless at any size, that well-tuned INT8 W8A8 loses between 1% and 3% of accuracy, and that weight-only INT4 is more competitive than expected, on par with 8 bits [10]. Lee et al., with models from 1 billion to 405 billion parameters and 13 datasets, observe that small models can suffer severe drops at 4 bits, that 70 billion models remain stable, that AWQ tends to outperform GPTQ in weight-only quantization, and that quantized models usually fail more at following instructions and detecting hallucinations [11]. Dettmers and Zettlemoyer, with more than 35,000 experiments, conclude that 4 bits is almost universally the optimal precision for the total bits of the model and its zero-shot accuracy: a 30 billion parameter model at 8 bits and another of 60 billion at 4 bits occupy the same space, but can perform very differently, so both the size of the model and the bits matter [9].

These figures should not be applied as is. Those from GPTQ, AWQ, and SmoothQuant come from 2022 and 2023 models; perplexity on WikiText does not measure instruction following or the absence of hallucinations; and differences between methods can change with each model family. Their utility is to fix an order of magnitude —8 bits almost lossless, 4 bits acceptable in large models, 3 bits or less only with specific methods— that your own test must confirm.

Summarized decision tree

First, define the operational contract. Write the maximum input length that must actually be supported, a reserve of output tokens, the maximum number of live requests, the latency target, and the critical tasks. Differentiate an exceptional maximum from a usual target. If an application processes long documents, measuring only brief messages does not represent its memory risk or its useful quality.

Then gather the parameters of the architecture and the runtime. For the model, record layers, KV heads, head dimension, and weight format. For the runtime, record the cache data type, whether it reserves static or dynamic cache, whether it allows offloading, the GPU memory limit, and any batch or in-flight token adjustment. In service tools, the cache budget can be explicitly configured or derived from a fraction of available memory; both cases must be noted in the experiment.

Calculate a band, not a single figure: observed or estimated weights, KV cache for the target load, a reserve for temporaries, and a margin. If the total exceeds available VRAM before applying the margin, discard the combination. If it fits by very little, classify it as a risk candidate and test it under maximum load. If it fits with margin, do not consider it approved until quality and latency are validated.

Choose at least three candidates that respond to different hypotheses: the desired model at 8 bits, 6 bits, and 4 bits; or, when a candidate does not make sense, a smaller model with higher precision. Keep base model, revision, prompt, maximum context, output limit, seed when compatible, decoding parameters, hardware, and runtime version constant. Changing several variables at once prevents attributing a difference to quantization.

Reproducible process in seven steps

1. Define input context, reserved output, concurrency, and target latency.

2. Record architecture, weight artifact, and cache configuration.

3. Estimate weights, KV cache, and margin for the maximum live tokens.

4. Discard candidates that do not fit before the margin or that require unverified assumptions.

5. Run comparable candidates with identical parameters.

6. Measure memory, time to first token, generation, errors, and output quality.

7. Keep the configuration only if it exceeds the quality threshold and maintains margin under maximum load.

A useful test does not need to be huge, but it must be representative. Build a small set of cases that includes the work motivating the deployment: structured extraction, classification, document synthesis, code assistance, or constrained responses, as appropriate. Include inputs of usual length and some close to the operational limit. Long inputs are necessary because they can reveal both memory errors and losses in instruction following or detail retrieval.

For each case, define what will be validated before running the model. Some tasks allow exact comparison: a valid JSON schema, allowed labels, mandatory fields, a query that must contain specific values, or automated tests for code. Others require human review with a rubric: fidelity to the document, coverage, absence of inventions, format compliance, and utility. Do not confuse fluency with correctness.

Set an explicit threshold. For example, a configuration can be discarded if it fails more critical cases than the reference candidate, if it worsens structural validity beyond a limit decided by the team, or if it produces new errors on sensitive data. The threshold belongs to the application's risk; it cannot be inferred from the bits. In a creative drafting task, more variation may be acceptable than in data extraction for a subsequent process.

Repeat the tests. With stochastic decoding, several runs help separate generation variation from systematic degradation. With deterministic decoding, repetitions are still useful to observe performance stability and memory errors. Report results by task type and input length, not just a global average. A quantization that seems equivalent on average may concentrate its failures in the longest documents or in the highest-impact task.

Fair comparison

Do not compare a quantization with a context of 4,000 tokens against another with 16,000, nor one with offloading and another without, if you want to attribute differences to the weights. If a condition is changed due to memory needs, report it as a different operational configuration.

Perplexity measures the average surprise of the model before a reference text; it is calculated with the original model and the quantized one on the same text, and the cited works report it on WikiText and C4 [5][6][18]. llama.cpp documentation recommends perplexity and KL divergence between the distributions of the original and quantized models as accuracy metrics for the quantized model [21]. Jin et al. observe that perplexity serves as a surrogate indicator in most of the benchmarks they tested [12], but Lee et al. show that instruction following and hallucination detection suffer more than plain text metrics suggest [11]. That is why perplexity serves to discard candidates and the minimum test with own cases remains the final decision.

Calibration is another variable. GPTQ calibrates with 128 random segments of 2,048 tokens from C4 [5] and AWQ, with 16 sequences [6]; llama.cpp offers an importance matrix (imatrix) to guide quantization [21]. Unverified: the sources consulted do not measure how calibrating with primarily English text affects quality in Spanish or other languages; check it with your corpus.

A practical warning from llama.cpp: requantizing a model that was already quantized, with the --allow-requantize option, produces severe quality loss [21]. Always start from 16-bit weights and note which specific artifact has been downloaded, in the minimum execution log.

On a laptop with limited GPU, the priority is usually to avoid a configuration that continuously depends on system RAM or offloading for an interactive experience. Start with a realistic context, a single request, and a model or quantization that leaves margin. If 4 bits is the only way to load the model, also compare a smaller model at 6 or 8 bits. The latter may be more stable and more useful in the specific task, even if it has fewer parameters.

On a workstation with one GPU, there is more margin to choose between quality, context, and speed, but the limit is still shared by weights, cache, and temporaries. It is an appropriate environment to compare 4, 6, and 8 bits under the same corpus and to decide whether to reserve VRAM for long contexts. If alternating between brief sessions and document analysis is planned, measure both profiles: the result of a short conversation does not size the second.

On a server with moderate concurrency, the planning unit is no longer the model file, but the total capacity of live tokens. The cache manager and request scheduler are part of the decision. A configuration that works for a single session can exhaust memory when long prefills coincide. Define admission limits, maximum length, output reserve, and concurrency; then test bursts and mixes of short and long requests. Queue metrics and latency percentiles are more informative than the best isolated result.

In any of the three profiles, offloading is an option that must be declared, not an invisible solution. It can expand apparent capacity by moving part of the data, but it can also alter latency and depend on the link between CPU and GPU. Decide with measurements whether that trade-off is acceptable for the use case.

Choosing 4 bits is not a decision about the model, but about the model, format, and engine trio. GGUF is a binary format that stores tensors and a standardized set of metadata, designed for llama.cpp and derived tools like LM Studio, GPT4All, and Ollama [22]. Other formats live in other engines, with their own hardware requirements. The table collects what their official pages consulted on October 17, 2026, document; this field changes fast and it is advisable to review the current version.

What each engine documents about quantization

Two consequences for the decision. The first: on hardware without the appropriate instructions, quantizing can worsen performance due to the cost of quantizing and dequantizing, as ONNX Runtime warns [3]. The second: the best format depends on the usage pattern. Kurtic et al. find that W4A16 is the most cost-effective with synchronous requests and that W8A8 dominates with continuous asynchronous batch processing [10]. It is the same distinction this article makes between the single-session laptop and the server with concurrency.

Note: base model identifier and revision; quantization artifact and method; runtime version and backend; GPU and available memory; KV cache type; context limit; output reserve; batch, concurrency, and in-flight token limit; offloading; decoding parameters; corpus and rubric; minimum free memory; time to first token; generation rate; errors and quality result. Without this log, a subsequent comparison will hardly be reproducible.

Discard a configuration when it produces intermittent memory errors, even if a brief demonstration works. Intermittency usually indicates that available memory depends on the shape of the requests, the prefill peak, fragmentation, or other process loads. It is also a reason for discard if the runtime silently reduces the effective context, if the system is only stable with a load lower than expected, or if the observed margin disappears in repeated tests.

Quality degradation must be analyzed by pattern. Errors concentrated in extraction, calculations, mandatory fields, instruction following, or extensive documents carry more weight than stylistic changes if those tasks are critical. An apparently reasonable output but with invented values should not be approved just by an average score. Also review format validity when the output feeds subsequent software.

Before setting a default option, check privacy and operation. Running inference on the device does not by itself guarantee that no data leaves it: model downloads, telemetry, prompt logging, dependency updates, and observability tools are separate aspects. What data is retained and what communications each component makes must be verified in the configuration and the chosen network environment.

The final result does not have to be "the largest possible quantization". It can be 6 bits for an interactive profile, 4 bits for an analysis profile with large context, or 8 bits for a task where the detected loss is not acceptable. Keep the alternatives along with their test log. If the runtime, hardware, weight format, or workload changes, measure again: the previous conclusion ceases to be a guarantee.

Checklist before adopting a quantization

• Weights, KV cache, temporaries, and margin have been measured or justified separately.

• Context, reserved output, and concurrency reflect the maximum expected load.

• It has been checked whether quantization affects weights, KV cache, or both.

• Compared configurations use the same base model and equivalent conditions.

• The corpus contains critical tasks and representative long inputs.

• There is an acceptable loss threshold defined before reviewing results.

• Minimum free memory, errors, and latency percentiles have been recorded.

• Offloading, telemetry, downloads, and logging configuration has been reviewed.

• The decision can be reproduced from the complete technical log.

The figures from GPTQ, AWQ, SmoothQuant, and LLM.int8() were measured on OPT, BLOOM, and Llama 2; they do not extrapolate to current models. Kurtic et al. and Lee et al. use Llama 3.x and are closer, but do not cover all models or languages.

The speed improvements of KIVI, KVQuant, and AWQ (TinyChat) come from the authors' kernels and GPU. In the llama.cpp README, the hardware of the speed measurements and the precision of the original size column are not indicated, although their figures (32.1 GB for 8B) match about 4 bytes per parameter; this reading is our inference and is not in the source.

The architecture parameters of the KV cache table come from public replicas of the config.json files on Hugging Face (NousResearch and unsloth), because the official Meta repositories require authentication. The head dimension of Llama 3.1 70B (128) is derived by dividing the hidden size by the number of attention heads.

The compatibility table reflects the documentation consulted on October 17, 2026, including the development version of vLLM. Unverified: the effect of calibration language, the behavior of each quantizer with mixture-of-experts architectures, and performance on your specific GPU; those three things require your own measurement.

• [1] Gholami, A. et al. (2021). A Survey of Quantization Methods for Efficient Neural Network Inference. arXiv:2103.13630. https://arxiv.org/abs/2103.13630

• [2] Wu, H. et al. (2020). Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation. arXiv:2004.09602. https://arxiv.org/abs/2004.09602

• [3] ONNX Runtime. Quantize ONNX models. https://onnxruntime.ai/docs/performance/model-optimizations/quantization.html

• [4] Dettmers, T., Lewis, M., Belkada, Y., Zettlemoyer, L. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. arXiv:2208.07339. https://arxiv.org/abs/2208.07339

• [5] Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv:2210.17323. https://arxiv.org/abs/2210.17323

• [6] Lin, J. et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. arXiv:2306.00978. https://arxiv.org/abs/2306.00978

• [7] Xiao, G. et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. arXiv:2211.10438. https://arxiv.org/abs/2211.10438

• [8] Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314. https://arxiv.org/abs/2305.14314

• [9] Dettmers, T., Zettlemoyer, L. (2022). The case for 4-bit precision: k-bit Inference Scaling Laws. arXiv:2212.09720. https://arxiv.org/abs/2212.09720

• [10] Kurtic, E. et al. (2024). «Give Me BF16 or Give Me Death»? Accuracy-Performance Trade-Offs in LLM Quantization. arXiv:2411.02355. https://arxiv.org/abs/2411.02355

• [11] Lee, J. et al. (2024). Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giant. arXiv:2409.11055. https://arxiv.org/abs/2409.11055

• [12] Jin, R. et al. (2024). A Comprehensive Evaluation of Quantization Strategies for Large Language Models. arXiv:2402.16775. https://arxiv.org/abs/2402.16775

• [13] Micikevicius, P. et al. (2022). FP8 Formats for Deep Learning. arXiv:2209.05433. https://arxiv.org/abs/2209.05433

• [14] Rouhani, B. D. et al. (2023). Microscaling Data Formats for Deep Learning. arXiv:2310.10537. https://arxiv.org/abs/2310.10537

• [15] Ma, S. et al. (2024). The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits. arXiv:2402.17764. https://arxiv.org/abs/2402.17764

• [16] Egiazarian, V. et al. (2024). Extreme Compression of Large Language Models via Additive Quantization. arXiv:2401.06118. https://arxiv.org/abs/2401.06118

• [17] Liu, Z. et al. (2024). KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. arXiv:2402.02750. https://arxiv.org/abs/2402.02750

• [18] Hooper, C. et al. (2024). KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization. arXiv:2401.18079. https://arxiv.org/abs/2401.18079

• [19] Kwon, W. et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. arXiv:2309.06180. https://arxiv.org/abs/2309.06180

• [20] Ainslie, J. et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. arXiv:2305.13245. https://arxiv.org/abs/2305.13245

• [21] ggml-org. llama.cpp, tools/quantize/README.md. https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md

• [22] Hugging Face. GGUF (Hub documentation). https://huggingface.co/docs/hub/gguf

• [23] Hugging Face. Quantization (Transformers documentation). https://huggingface.co/docs/transformers/main/en/quantization/overview

• [24] Hugging Face. bitsandbytes (Transformers documentation). https://huggingface.co/docs/transformers/main/en/quantization/bitsandbytes

• [25] Hugging Face. KV cache strategies (Transformers documentation). https://huggingface.co/docs/transformers/en/kv_cache

• [26] Hugging Face. Optimizing LLMs for speed and memory. https://huggingface.co/docs/transformers/main/en/llm_tutorial_optimization

• [27] vLLM. Quantization. https://docs.vllm.ai/en/latest/features/quantization/

• [28] NVIDIA. TensorRT LLM: Quantization. https://nvidia.github.io/TensorRT-LLM/features/quantization.html

• [29] PyTorch. Quantization (2.14 documentation). https://docs.pytorch.org/docs/2.14/quantization.html

• [30] Hugging Face. config.json files for Llama 2 7B, Llama 3 8B, Llama 3.1 8B and 70B (NousResearch and unsloth mirrors). https://huggingface.co/unsloth/Meta-Llama-3.1-8B/blob/main/config.json

• [31] Liu, Z. et al. (2023). LLM-QAT: Data-Free Quantization Aware Training for Large Language Models. arXiv:2305.17888. https://arxiv.org/abs/2305.17888