25 Oct 2023 AI Engineering 5 min read

Fine-Tuning GPT-3.5 Turbo vs. RAG: Form Is Trainable, Knowledge Is Not

OpenAI opened GPT-3.5 Turbo fine-tuning on August 22, 2023. We define what fine-tuning actually improves — output format, tone, narrow tasks — and what it does not: adding knowledge. Using the October 2023 price list we calculate training cost, the eightfold inference premium and the break-even, and show why retrieval-augmented generation remains the correct channel for facts.

Two Levers for Adapting a Language Model

On August 22, 2023, OpenAI opened fine-tuning for GPT-3.5 Turbo. Within days we received the same request from several directions: "Fine-tune the model on our documentation so it knows our products." The request is understandable. It is also based on a misconception. Fine-tuning is not a mechanism for adding knowledge to a model. Applied to the wrong job, it costs money and delivers hallucinations.

There are two levers for adapting a hosted language model. Fine-tuning changes the model's weights through supervised training on example conversations. Retrieval-augmented generation (RAG) changes the model's input: relevant text is fetched at query time and placed in the context window. The two levers solve different problems. This article defines the boundary and puts numbers on it.

A request arrives — the router classifies its difficulty. 1/4

What OpenAI Shipped on August 22

The release covers supervised fine-tuning for gpt-3.5-turbo-0613 via the API. You upload training data as JSONL chat transcripts, start a fine-tuning job, and receive a private model checkpoint that you address by its own name. The context length during fine-tuning is 4,000 tokens; support for the 16k variant is announced for later this fall, as is fine-tuning for GPT-4.

Two operational details matter. Training data passes through OpenAI's moderation API and a GPT-4-based moderation system before training starts. And OpenAI states that data sent through the fine-tuning API is owned by the customer and is not used to train other models. OpenAI's headline claim: a fine-tuned GPT-3.5 Turbo can match or even outperform base GPT-4 on certain narrow tasks.

What Fine-Tuning Actually Improves

Fine-tuning shapes the output distribution. It is effective in exactly three categories. First, format reliability: a model fine-tuned on valid JSON responses produces valid JSON far more consistently than one that is merely instructed to. Second, tone and style: a support bot that must write in a defined brand voice, in German, at a fixed register. Third, narrow tasks: classification, extraction, or completion against one fixed schema, where the task distribution is stable.

A fourth effect is prompt compression. Instructions and few-shot examples that occupy hundreds of tokens per request can be trained into the weights. OpenAI reports that early testers reduced prompt sizes by up to 90 percent. Shorter prompts mean lower latency. Whether they mean lower cost is a calculation, not an assumption. We do that calculation below.

What Fine-Tuning Does Not Do

Fine-tuning does not reliably add knowledge. A few thousand training examples adjust how the model responds; they do not build a queryable store of facts. Facts learned this way have no provenance: the model cannot state where an answer came from, and it interpolates plausible falsehoods where its weights are silent. What was actually retained can only be established by testing — per fact.

The operational problems weigh heavier than the statistical ones. Weights are stale the moment a document changes; the fix is a new training run. There is no per-user access control inside a checkpoint: every fact you train in is available to every caller of that model. And fine-tuning extends neither the context window nor reasoning capability. It narrows behavior; it does not broaden competence.

The Cost Math in October 2023

Training is cheap. Fine-tuning costs $0.008 per 1,000 training tokens; OpenAI's own example prices a 100,000-token training file over three epochs at roughly $2.40. Inference is not cheap. A fine-tuned GPT-3.5 Turbo costs eight times the base model per token.

The break-even follows directly. At an eightfold per-token premium, prompt compression pays off only if you remove more than 87.5 percent of your tokens — the upper edge of what OpenAI reports. Against base GPT-3.5, fine-tuning rarely wins on cost. The realistic case is a different one: replacing GPT-4 on a narrow task. There, fine-tuned GPT-3.5 costs 60 percent less on input and 73 percent less on output, at lower latency.

Model	Input per 1K tokens	Output per 1K tokens	Training per 1K tokens
GPT-3.5 Turbo (4K)	$0.0015	$0.0020	—
GPT-3.5 Turbo fine-tuned	$0.0120	$0.0160	$0.0080
GPT-4 (8K)	$0.0300	$0.0600	—

Retrieval Is the Knowledge Channel

Retrieval-augmented generation was described by Lewis et al. in 2020: retrieve documents relevant to a query and condition generation on them. The engineering version in 2023 is straightforward. Split the corpus into chunks, embed each chunk with text-embedding-ada-002, store the vectors, and at query time insert the top-ranked chunks into the prompt.

The economics sit on a different scale. Embeddings cost $0.0001 per 1,000 tokens; a corpus of 10 million tokens embeds for one dollar. Updates are index writes, effective in seconds. Every answer can cite its source chunks. Access control lives in the retrieval layer — where it belongs. The failure modes are honest ones: bad chunking, weak ranking. Both are measurable.

Fine-Tune for Form and Retrieve for Facts

The decision rule we apply in client projects: fine-tune for form, retrieve for facts. If the failure reads "wrong format, wrong tone, ignores instructions," fine-tuning is the correct tool. If it reads "does not know X" or "X changed last week," retrieval is the correct tool. Most production systems that look like they need fine-tuning need retrieval plus a better prompt.

The combination is legitimate and underused. Fine-tune GPT-3.5 Turbo on examples of answering from provided context — refusing when the context does not contain the answer, citing chunks, holding the output schema. Let RAG supply the context. One precondition applies to both levers: an evaluation set before any training run. Without one, you cannot tell whether the fine-tune helped or merely changed the flavor of your errors.

Outlook From October 2023

From where we stand in October 2023, the trajectory looks like this. Fine-tuning for GPT-4 is announced for this fall, and a fine-tuning UI is planned. Per-token prices have fallen steadily since March; we expect the inference premium for fine-tuned models to shrink. That weakens the cost argument — not the knowledge argument. That one is structural.

We also expect base models and function calling to absorb more of the format work that requires fine-tuning today, while retrieval tooling — vector databases, chunking, re-ranking — matures into standard infrastructure. Our prediction: within a year, "should we fine-tune?" will be a routine architecture-review question with a short checklist answer, and the default answer will still be retrieval first.