AI log, day 01: tokens, embeddings, and how a model actually generates text

I have written a handful of posts about AI from the outside: what it changed on my team, what it means for people who never touch a keyboard for work, whether the doom predictions hold up. What I have not done is sit down and actually learn how the thing works, the way I’d learn a new language or a new database. So I am starting a log. Day 01, day 02, and so on, written in public, alongside the usual backend and career posts, which keep going as normal.

The rule I am setting for myself: no topic counts as learned until I can explain it the way I’d explain a bug to another engineer, not the way a marketing page explains it. I am not starting from zero. I have shipped a RAG pipeline already, so I know roughly what an embedding and a vector search are for. What I want out of this log is the layer underneath that: why those pieces are shaped the way they are, not just how to call the library that implements them. So day one is longer than “here’s what a token is” would suggest, it walks through four ideas that everything else in this field sits on top of, then comes back to my own RAG project with that vocabulary in hand.

A token is not a word

The first wrong assumption I had to drop: a language model does not read words, and it does not read letters either. It reads tokens, which are chunks somewhere in between, produced by a tokenizer before any of the actual model runs. Common whole words usually get their own token. Rarer words, made-up words, and most non-English text get split into two or more pieces.

That splitting is why a sentence that looks short to you can be expensive, and a sentence that looks long can be cheap. “the” is one token. A word like “tokenization” is likely two, something like “token” and “ization”. A name like “Hetauda” almost certainly splits into pieces too, because the tokenizer’s vocabulary was built by scanning huge amounts of mostly-English internet text and keeping whatever chunks showed up often enough to earn their own slot. Common English fragments get cheap, single-token treatment. Everything else pays a small tax in extra tokens.

OpenAI’s own rule of thumb has stayed roughly the same for years: about 4 characters per token, or roughly three-quarters of a word, for typical English text. It is an average, not a guarantee, and it moves a lot once you’re outside English or writing something dense with numbers, code, or unusual proper nouns.

Every API that serves these models bills per token, both the tokens you send in and the tokens you get back. That reframes a few things I used to think about loosely: a system prompt is a fixed per-request cost paid on every single call, the same way a duplicated database query is paid on every request even if the result never changes, and a context window limit is a token count, not a word count, so “how much can I paste in” depends on what you’re pasting, not how long it looks on screen.

You can see the split yourself without touching a model at all, using the tokenizer OpenAI publishes as the tiktoken library:

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4")
tokens = enc.encode("Debugging a goroutine leak in production")
print(len(tokens), tokens)

Run that on a plain English sentence built from common words and the token count sits close to the word count. Run it on a sentence full of proper nouns, code identifiers, or non-English text, and the count climbs, because more of the input is falling back to smaller, less common chunks.

Embeddings: turning tokens into a position in space

Tokens are still just symbols, a model can’t do arithmetic on the string “goroutine” directly. An embedding is what fixes that: a model maps a chunk of text to a vector, a list of a few thousand floating point numbers, such that pieces of text used in similar contexts end up close together in that space, and unrelated ones end up far apart. Model providers publish the exact size: OpenAI’s text-embedding-3-small produces 1536 numbers per input, text-embedding-3-large produces 3072. Neither of those numbers means anything on its own, what matters is the distance between two vectors, usually measured with cosine similarity, a value that says how close two pieces of text are in meaning without either one containing the other’s exact words.

This is the part I had already used without fully understanding it. My RAG pipeline embedded every chunk of a runbook or postmortem once, stored the resulting vectors, and at query time embedded the incoming question and searched for the nearest stored vectors. I treated that search as a slightly fancier LIKE '%keyword%'. It’s actually closer to asking “which of these stored points in a several-thousand-dimensional space sits closest to this new point”, which is why it can match a question phrased completely differently from the document and still find the right chunk, and also why it can confidently return the wrong chunk when two unrelated topics happen to use similar language.

Context windows are a token budget, not a document limit

A model has a fixed maximum number of tokens it can hold across one request, input and output combined, called the context window. It is easy to think of it as “how many pages can I paste in”, but it is a token count, so the same limit holds far less text once that text is dense with rare tokens.

Two things about context windows changed how I think about RAG specifically:

• Every chunk you retrieve and stuff into the prompt is billed as input tokens on every single call, so a bigger context window doesn’t mean it’s free to use more of it.
• Position inside the context matters, not just total size. Research on long-context models has repeatedly found that a model is more reliable about information near the start or end of its input than something buried in the middle, an effect usually called “lost in the middle.” A bigger window that gets careless use can quietly get less reliable, not more capable.

That’s the actual reason RAG retrieves a handful of relevant chunks instead of just dumping an entire knowledge base into the prompt. It’s not only that the whole knowledge base wouldn’t fit, it’s that cost and reliability both get worse long before the window itself runs out.

Generation: predicting one token at a time

The last piece is how a model turns all of that into an answer. It does not draft a full response and hand it over. It predicts a probability distribution over “what token comes next”, picks one, appends it to its own input, and repeats, one token at a time, until it decides to stop. This is why it’s called autoregressive generation, each new token is generated conditioned on everything before it, including tokens the model itself just wrote a moment ago.

Two things follow directly from that mechanism. First, there’s no backtracking. Once the model has committed to a token that starts down a wrong path, it keeps building fluently on top of that mistake instead of noticing and correcting it, the same way autocomplete confidently keeps finishing a sentence you didn’t mean to start. Second, this is the actual, mechanical source of hallucination: the model is doing next-token likelihood prediction, not fact lookup, so a fluent, plausible-sounding, wrong answer costs it nothing extra to produce compared to a correct one, unless something in its input is actually anchoring it to real information.

That last clause is exactly what retrieval is for.

What my RAG pipeline was actually doing

Putting all four ideas together changes how I’d describe the project I already shipped. At index time, each runbook and postmortem got split into chunks and each chunk got embedded, once, into a vector stored for later search. At query time, the incoming question got tokenized and embedded the same way, and a nearest-neighbor search over the stored vectors pulled back the chunks whose meaning sat closest to the question, not the ones sharing the most literal words. Those chunks got inserted into the prompt, spending part of the context window’s token budget, positioned deliberately rather than dumped in arbitrarily, given what I now know about the middle of a long context being the least reliable spot. Then generation ran token by token, grounded in that retrieved text instead of only in whatever the model happened to memorize during training, which is the whole reason a RAG answer can cite a real internal document instead of confidently inventing one.

When I built it, chunk size and embedding model felt like defaults to accept from a tutorial. Now I can actually reason about why they matter: a chunk too large wastes context tokens and dilutes what the embedding represents, a chunk too small loses the surrounding meaning that made it findable in the first place, and the embedding model’s training data determines what “similar” even means for your specific documents.

What’s next

Tokens, embeddings, context windows, and generation are the floor everything else stands on. From here the log moves to the knobs and decisions built on top of that floor: temperature and sampling, prompting as an interface rather than a trick, when fine-tuning actually beats a bigger prompt or more retrieval, how to evaluate whether a change actually helped, what an “agent” is once you strip the buzzword away, and a closer look at hallucination now that the mechanism behind it is on the table.

Regular posts continue in between. This is a side log, not a replacement for the rest of what I write here.