LoRA and PEFT in Practice
A practical guide to parameter-efficient fine-tuning as of April 2024. We cover how LoRA works, what QLoRA changes with 4-bit quantization, the current open-source toolchain, and a decision framework for tuning versus retrieval versus prompting — including what adapters do not deliver and how thousands of them run on one GPU.
The Cost Problem of Model Adaptation
Full fine-tuning updates every parameter of a model. For a 65-billion-parameter model in 16-bit precision this requires more than 780 GB of GPU memory — Dettmers et al. measured this for LLaMA 65B. Each adapted variant is a complete copy of the weights. Ten tasks mean ten copies of the full model.
Open weights are now good enough to matter: Llama 2 (July 2023), Mistral 7B (September 2023), Mixtral 8x7B (December 2023), Llama 3 (18 April 2024). The practical question for engineering teams is how to specialize these models without a GPU cluster and without maintaining a fleet of 140 GB checkpoints.
How LoRA Works
LoRA (Hu et al., 2021) freezes the pretrained weights and injects trainable low-rank matrices into selected layers. Instead of learning a full weight update ΔW, it learns the decomposition ΔW = BA, where the rank r is small — often between 1 and 16. Only A and B are trained. The base model never changes.
The numbers from the original paper are concrete. On GPT-3 175B, LoRA reduced trainable parameters by a factor of 10,000 and GPU memory during training by a factor of 3. The task checkpoint shrank from 350 GB to 35 MB at rank 4. After training, the adapter can be merged into the base weights. Merged, LoRA adds zero inference latency. Quality matched or exceeded full fine-tuning on the reported benchmarks.
Where the adapters go matters. The paper found that adapting the query and value projections together outperformed spending the same parameter budget on a single matrix type at higher rank. Rank 4 on Wq and Wv was sufficient for GPT-3; larger ranks brought no consistent gain.
QLoRA Brings Tuning to a Single GPU
QLoRA (Dettmers et al., May 2023) combines LoRA with a 4-bit quantized base model. Gradients are backpropagated through frozen 4-bit NormalFloat (NF4) weights into the adapters. Two supporting techniques — double quantization and paged optimizers — cut memory further. The result: fine-tuning a 65B model drops from over 780 GB to under 48 GB. That is a single professional GPU.
The paper's Guanaco 65B model reached 99.3% of ChatGPT's level on the Vicuna benchmark after 24 hours on one GPU. That benchmark measures chat preference, not factual correctness — a narrow claim. The durable result is different: 4-bit NF4 QLoRA matched 16-bit fine-tuning on academic benchmarks. In practice this puts 7B to 13B models within reach of a single 24 GB card.
The Toolchain in Spring 2024
The standard stack is Hugging Face PEFT (released 10 February 2023) for the adapter methods, bitsandbytes for 4-bit quantization, and TRL for supervised fine-tuning and DPO. Axolotl wraps the same components behind a YAML configuration. The knobs that matter are few: rank, alpha, target modules, learning rate, and epochs.
Data quality dominates all of them. The QLoRA study found that roughly 9,000 curated OASST1 samples produced a better chatbot than a 450,000-sample subset of FLAN v2. Our experience matches this: a small clean dataset with consistent formatting beats a large noisy one. Budget more time for data than for training.
Tuning versus Retrieval versus Prompting
Fine-tuning is one of three tools, and it is the most expensive one to operate. The default order is: prompting first, retrieval second, parameter-efficient tuning third. Fine-tuning earns its place when the model must change behavior — output format, tone, domain vocabulary, a narrow task performed thousands of times — not when it must know more.
The costs differ in kind. Prompting costs latency and tokens on every request. Retrieval costs infrastructure: an index, a chunking strategy, freshness pipelines. Tuning costs upfront: data curation, training runs, evaluation, and a model artifact you now own and must maintain. Pick the cheapest tool that meets the requirement — usually in that order.
| Approach | What changes | Best for | Main limitation |
|---|---|---|---|
| Prompting | Nothing — instructions in context | Fast iteration, low volume | Long prompts cost tokens on every call |
| Retrieval (RAG) | Nothing — facts injected at runtime | Current, verifiable knowledge | Retrieval quality caps answer quality |
| LoRA / QLoRA | Small adapter weights (MBs) | Format, style, domain behavior | Does not reliably add new facts |
What LoRA Does Not Do
LoRA does not reliably inject new knowledge. An adapter trained on your documentation will imitate its style; it will still hallucinate its contents. For facts, retrieval remains the correct tool. LoRA also cannot add capabilities the base model lacks — a 7B model that cannot reason through a task will not learn it from 5,000 examples.
Two operational limits. First, an adapter merged into the base weights serves exactly one task; the original paper notes that batching different tasks in one forward pass requires keeping adapters unmerged. Second, regressions on general capabilities are possible and must be measured — every fine-tune needs its own evaluation set before deployment.
Serving Many Adapters on One GPU
Because adapters are megabytes, one base model can host many of them. S-LoRA (Sheng et al., November 2023) demonstrated serving 2,000 LoRA adapters on a single GPU, using unified paging to manage adapter weights and KV cache in one memory pool. Throughput was up to 4x higher than vLLM with naive LoRA support and up to 30x higher than swapping adapters through PEFT.
This changes the economics of customization. Per-customer or per-task models no longer mean per-customer GPUs. The deployment unit becomes the adapter: versioned, a few dozen megabytes, hot-swapped against a shared base. We treat adapters as build artifacts with the same review and rollout discipline as code.
Outlook From April 2024
Llama 3, released on 18 April 2024, moves the open baseline again: an 8B model now handles tasks that needed 70B a year ago. We expect tuned 8B-class models to become the default for narrow production tasks, with QLoRA runs measured in hours and single-digit euros of compute.
Method development has not stopped: DoRA (February 2024) and GaLore (March 2024) refine what low-rank training can do. Our prediction for the next twelve months: multi-adapter serving becomes a standard feature of inference stacks, fine-tuning specializes in behavior while retrieval owns knowledge, and most production systems quietly combine prompting, retrieval, and one small adapter.
Sources
- Hu et al. — LoRA: Low-Rank Adaptation of Large Language Models (arXiv, 17 June 2021)
- Hugging Face — PEFT: Parameter-Efficient Fine-Tuning of Billion-Scale Models on Low-Resource Hardware (10 February 2023)
- Dettmers et al. — QLoRA: Efficient Finetuning of Quantized LLMs (arXiv, 23 May 2023)
- Sheng et al. — S-LoRA: Serving Thousands of Concurrent LoRA Adapters (arXiv, 6 November 2023)
- Meta AI — Introducing Meta Llama 3 (18 April 2024)