AI log, day 04: fine-tuning is the last resort, not the first move

Day 03 ended on a promise to cover fine-tuning versus RAG versus just writing a bigger, better-structured prompt. I went in expecting a real trade-off between three roughly comparable techniques, the kind of decision where you weigh pros and cons and pick based on taste. What I actually found is closer to a strict ordering, not a menu. Almost every problem that looks like it needs fine-tuning is solved first, and often solved completely, by treating the prompt as the interface it is (day 03’s whole point) or by handing the model fresh facts through retrieval. Fine-tuning is real and sometimes necessary, but it’s the last tool to reach for, not the first, and the industry’s habit of treating it as the “serious” option is mostly vibes.

What each one actually changes

The three options get compared constantly, but they’re not solving the same problem, which is exactly why the comparison feels slippery.

A bigger, better-structured prompt changes nothing about the model. It changes what’s in the context window for this one call: clearer instructions, few-shot examples, a forced output schema, the whole toolkit from day 03. Nothing persists past the call.

RAG (retrieval-augmented generation, the thing I built a whole pipeline around) also changes nothing about the model. It changes what gets fetched and inserted into the prompt before that call happens. The model still knows nothing new after the call ends. What changed is that the right facts, pulled from a document store, showed up in its context this time.

Fine-tuning is the only one of the three that changes the model itself. You take a base model’s weights and keep training them on examples specific to your task, and the model comes out the other side actually different, with new default behavior baked in rather than facts handed to it per call.

Said plainly: prompting and RAG are both about what you put in front of the model at request time. Fine-tuning is about changing what the model is. That’s a much bigger, slower, more expensive lever, and the size of that lever is exactly why it should be the last one pulled, not the first.

The question that actually decides it

Every explainer I’d half-absorbed before this log framed the decision as “how much data do you have” or “how much budget.” Neither turned out to be the right first question. The right first question is: is the problem missing knowledge, or missing behavior?

Missing knowledge looks like: the model doesn’t know your company’s refund policy, doesn’t know what shipped in last week’s changelog, doesn’t know the contents of a contract nobody trained it on. That’s not a defect in the model, it’s a gap between what it was trained on and what you need it to answer about. RAG closes exactly that gap, and it closes it in a way fine-tuning structurally can’t match: retrieval is updated by changing a document, today, for free. Fine-tuning “teaches” facts by baking them into weights, which means every policy update, every new document, every price change requires re-training to stay current. Using fine-tuning to keep a model current on facts is like recompiling a binary every time a config value changes. It technically works and it is the wrong layer for the job.

Missing behavior looks like: the model knows the facts fine, but it won’t stop writing three paragraphs when you need one line, or it won’t reliably adopt a specific tone, or it keeps drifting off a strict output format despite the schema. That’s not a facts problem, and more retrieval won’t touch it, because the facts were never what was missing. This is where fine-tuning actually earns its keep, and specifically only after the cheaper behavioral fixes from day 03, tight instructions, few-shot examples, forced schemas, have been tried and have genuinely stopped moving the needle.

Why the cheap options usually win before it gets that far

The reason the ordering matters isn’t ideology, it’s that the cheap options solve more of the “missing behavior” bucket than it looks like they should. A big chunk of what people reach for fine-tuning to fix turns out to be an under-specified prompt wearing a trench coat. Day 03’s example, a one-paragraph prompt doing four jobs at once with a hoped-for output format, produces exactly the symptom people describe as “the model won’t follow instructions,” and the actual fix was separating system message from instruction from data from schema, not retraining anything. Few-shot examples close most of the remaining gap, because a demonstration pins down format and tone far more precisely than a prose description ever will.

Cost is the other half of the ordering, and it isn’t subtle. A better prompt costs the time to write it and iterate on it, called on every request at normal inference price. RAG costs a document store, an embedding pipeline, and retrieval latency, real infrastructure but infrastructure that scales with your documents, not with model size. Fine-tuning costs a training run, a dataset of examples good enough to actually shift behavior in the right direction (which is its own hard problem, bad fine-tuning data teaches the model your mistakes as confidently as your successes), and a new model artifact to version, monitor, and eventually re-run training against when the base model itself gets upgraded. That last part is easy to miss: a RAG pipeline gets the benefit of a better underlying model the moment you swap the API call. A fine-tuned model is now its own artifact, frozen at whatever base model it was trained from, and upgrading means fine-tuning again.

Where fine-tuning genuinely wins

None of this makes fine-tuning wrong, it makes it specific. It’s the right call when the thing you need baked in is a persistent behavior at a volume where per-request prompt overhead actually costs real money, a consistent voice or format across millions of calls where re-sending the same few-shot examples every time is wasteful compared to a model that just does it by default. It’s also the right call when the pattern you need genuinely can’t be taught by example in a prompt at all, a narrow domain style, a classification boundary that’s fuzzy in ways no schema can express, something closer to muscle memory than to a fact you could look up. Those are real cases. They’re just a smaller slice of “the model isn’t doing what I want” than the term’s popularity suggests.

What I’d actually change now

The honest version of my own instinct going into this: I’d have reached for fine-tuning too early on a task that just needed a cleaner system prompt, because fine-tuning sounds like the rigorous engineering choice and a better prompt sounds like a hack. Backwards. The actual rigor is diagnosing which bucket the failure is in first, missing knowledge or missing behavior, then working up from the cheapest fix that bucket has, prompt structure before retrieval, retrieval before retraining, and only pulling the expensive lever once the cheap ones have been tried and specifically failed, not just skipped because they felt less serious.

What’s next

Day 05 tackles evals: how you actually tell whether a prompt or pipeline change made things better, instead of it just feeling better on the one example you happened to test by hand.

Regular posts continue in between, as always.