AI log, day 02: temperature and sampling, what turning the dial actually changes

Day 01 ended on generation: a model predicts a probability distribution over every possible next token, then picks one, appends it, and repeats. What I glossed over is the “picks one” step. That’s not one mechanism, it’s a knob, and it’s the knob almost every API exposes directly as temperature, top_p, and top_k. I’d been setting those to whatever a tutorial used without asking what they actually did to the numbers underneath. Turns out the answer is small enough to fit in a few lines of code, and specific enough that I can point at a decision in my own RAG pipeline and say it was set wrong.

Greedy decoding: always taking the best guess

The simplest possible policy is to always pick the single highest-probability token. That’s called greedy decoding, and it’s what you get from temperature=0. It’s fully deterministic: same input, same output, every time, because there’s no randomness in the choice at all.

That sounds like it should be the default for everything, and for some tasks it is. But a model that always takes the single most likely next token tends to produce flat, repetitive text, and worse, it has no way to recover from a token that was merely the safest choice rather than the best one two steps later. Greedy decoding optimizes each step in isolation; it never looks ahead. For open-ended writing that reads as boring and looping. For anything where you want exactly one correct, checkable answer, boring is precisely the point.

Temperature: reshaping the distribution before you sample

Every other setting starts from the model’s raw output scores, called logits, one number per token in the vocabulary, not yet a probability. Softmax turns logits into a probability distribution that sums to 1. Temperature is a number you divide every logit by before that softmax runs:

import math

def softmax_with_temperature(logits, temperature):
    scaled = [logit / temperature for logit in logits]
    peak = max(scaled)
    exps = [math.exp(x - peak) for x in scaled]
    total = sum(exps)
    return [e / total for e in exps]

logits = [4.1, 3.9, 1.2, 0.3]  # scores for 4 candidate next tokens
print([round(p, 3) for p in softmax_with_temperature(logits, 0.2)])
print([round(p, 3) for p in softmax_with_temperature(logits, 1.0)])
print([round(p, 3) for p in softmax_with_temperature(logits, 2.0)])

Run that and the pattern is obvious. At 0.2 the top two logits, already close to each other, get pulled almost all the probability mass, the other two are pushed near zero. At 1.0 you get back the model’s actual distribution, unscaled. At 2.0 the gap between the best token and the rest shrinks, and tokens that were barely plausible get a real shot at being picked. Low temperature sharpens the distribution toward whatever already looked strong. High temperature flattens it, giving weaker candidates a real chance. Temperature never adds new candidates and never removes any, it only changes how concentrated the probability is across the ones the model already considered.

Sampling itself is just: draw one token from whatever distribution comes out the other end of that scaling. temperature=0 is the limit case where the scaling is so extreme that sampling collapses back into greedy decoding, which is why the two are usually described together.

Top-p and top-k: cutting the tail instead of reshaping it

Temperature never removes a token from consideration, it just changes its odds. Top-p (nucleus sampling) and top-k are the settings that actually cut candidates out before sampling happens.

Top-k keeps only the k highest-probability tokens and throws away the rest before sampling from what’s left. It’s a blunt, fixed-size cutoff: the same k whether the model was genuinely torn between 40 plausible tokens or dead certain about one.

Top-p is the more adaptive version: sort tokens by probability, add them up until the running total crosses p, and keep only that set. When the model is confident, that set might be two or three tokens even with a generous p. When it’s uncertain, the same p might keep dozens. That’s the behavior I actually want most of the time, the cutoff should track how sure the model is, not sit at a fixed count regardless.

The three settings stack, they don’t replace each other. Temperature reshapes the whole distribution, then top-p or top-k trims the reshaped distribution down to a candidate set, then sampling draws from what’s left. Setting all three aggressively at once, low temperature and a tight top-p, doesn’t compound into something more precise, it mostly just converges back toward greedy decoding through two different doors.

Where this matters more than it looks like it should

None of this is exotic. It’s the exact reason the same prompt against the same model can return a different answer on two consecutive calls, which is a genuinely confusing thing to hit for the first time if you’re used to thinking of software as deterministic given fixed input. It’s also why “it worked when I tried it” is a weak signal on its own: at any temperature above zero, one successful run doesn’t tell you what the failure rate looks like across a hundred runs of the same prompt.

That has a direct, practical split. Anything where there’s one correct answer and you can check it, structured data extraction, classification, generating code that has to compile, calling a function with the right arguments, wants temperature=0 or something close to it. Determinism is a feature there, not a limitation. Anything where there’s a wide range of acceptable answers and variety is actually valuable, brainstorming, varying marketing copy, casual conversation, is where a higher temperature and a real top-p earn their keep.

What I’d change in my RAG pipeline

Going back to the RAG pipeline I keep returning to in this log: I left the generation call at the library’s default temperature, which for most chat APIs sits around 1.0, because I never had a reason to think about it. Knowing what I know now, that was the wrong default for that specific job. The whole point of retrieving chunks and stuffing them into the prompt was to ground the answer in real text instead of the model’s own memorized guesses. Leaving temperature at a setting tuned for open-ended variety adds randomness on top of a task that’s supposed to behave like lookup-plus-summarization, not creative writing. A grounded question-answering endpoint is exactly the kind of “there’s a right answer, and I can check it” task from the last section. It should have been running close to temperature=0, with a tight top-p as backup, from day one.

What’s next

Next entry moves to prompting itself: treating the prompt as an engineering interface with real inputs and contracts, not a trick you stumble into by rephrasing something three times until it works. After that: fine-tuning versus RAG versus a bigger prompt, and how you’d actually measure whether any of these changes made things better instead of just feeling better.

Regular posts continue in between, as always.