How LLMs Work: A Practitioner's Mental Model

Build just enough intuition about tokens, context windows, and next-token prediction to reason confidently about why LLMs behave the way they do — without the math.

Estimated time: ~12 min
Difficulty: intro
Sources: 6 sources

You paste 10,000 words of documentation into a chat, ask a question about paragraph 3, and the model answers perfectly. Then you ask it to recall your name from your very first message — and it draws a blank. The model isn’t being forgetful. It’s doing exactly what it was built to do. Once you see the mechanism, every quirk snaps into focus.

Text isn't what the model sees

Before the model processes a single word, your text goes through tokenization. A token is not a word, not a character, and not a syllable — it is whatever string fragment the training process found useful to reuse frequently across a vast corpus of text.

Common short words like the and is are single tokens. Longer or unusual words get fractured into sub-word pieces. Spaces are often fused into the following token, so brown (with a leading space) is typically one token, not two.

Surprising splits

The word unhelpfulness doesn’t stay whole — the tokenizer produces something like un + help + ful + ness: four tokens from one word. Meanwhile the is one token, THE is a different one, and import (because it appears constantly in code) gets its own token. Numbers are expensive: 123456 can be six tokens, one per digit — the model never saw that exact number often enough to give it a single slot.

Type any phrase below and watch the fragments appear:

Each colored block is one token. Leading spaces fuse into the following word (shown as ·).

What tokenization means in practice

Three consequences every practitioner should internalize:

Token count is not word count. English prose runs roughly 1.3 tokens per word. Code, markdown, and JSON run higher. API pricing is per token — not per word. ^{[tiktoken — OpenAI tokenizer]}
The model has no notion of individual characters. Asking it to count letters or reverse a word is hard because individual letters are not the unit it works in.
Rare words cost more. A technical term that barely appears in training data may expand to 6–8 tokens, eating your context budget and making the term harder for the model to reason about coherently.

Check your understanding

Why is counting the letter 'r' in 'strawberry' genuinely difficult for an LLM?

One very good guess, repeated

Next-token prediction def.

Given the sequence of tokens seen so far (the context), the model outputs a probability distribution over every token in its vocabulary (~50,000–100,000 entries). One token is sampled from that distribution, appended to the context, and the loop repeats until a stop condition is hit.

This is the entire generation mechanism. Not reasoning, not memory, not retrieval — just: given what came before, what is statistically likely to come next? Every feature you have ever seen in an LLM — code generation, essay writing, multi-step problem solving — is scaffolding built on top of this single loop.

Analogy — phone autocomplete is like LLM generation

Your phone’s autocomplete learned that “happy” often follows “I am” and surfaces it as the top suggestion. An LLM does the same thing over a vocabulary of roughly 100,000 tokens, trained on a substantial fraction of all human writing. Its “what comes next” instinct has absorbed enormous implicit world knowledge — which is both its power and the structural source of hallucination.

^{[tiktoken — OpenAI tokenizer]}

The widget below shows the model’s probability distribution before a token is sampled. Temperature controls how peaked or flat that distribution is — drag it and watch the bars shift:

Each bar is one candidate token. Raise temperature to flatten the distribution; lower it to concentrate probability on the top choice.

What temperature means

Temperature near 0: always pick the highest-probability token. Near-deterministic — good for factual tasks where you want repeatability.
Temperature = 1: sample from the raw learned distribution. The model’s “natural” voice.
Temperature above 1: flatten the distribution — lower-probability tokens become competitive. Useful for creative variation; dangerous for factual accuracy.

Common misconception

The LLM looks up the right answer in a database and writes it out.

What's actually true

There is no retrieval step in a standard LLM. The model has no key-value store queried at inference time. Everything it “knows” is encoded in billions of floating-point weights shaped during training. When it produces a confident-sounding fact, it is completing a high-probability pattern — not citing a source. Hallucination is a structural property of the mechanism, not a bug to be patched away.

Check your understanding

An LLM produces the correct capital city in response to 'The capital of France is'. What most likely happened?

The context window: a fixed-size spotlight

Every call to an LLM is stateless. The model has no persistent memory. What looks like “remembering” your earlier messages is your application re-sending the full conversation history on every request, packed into the context window.

Context window def.

The maximum total number of tokens the model processes in a single forward pass — input plus output combined. Once the window is full, tokens that no longer fit are simply absent. Not compressed, not summarized — absent.

Think of it as a spotlight of fixed width moving along a scroll. Whatever falls inside the spotlight is fully visible. Whatever scrolled off the left edge does not exist from the model’s point of view. ^{[Language Models are Few-Shot Learners]}

Fill the window with the sliders below and see exactly when things overflow:

Drag the sliders to fill the window. When total tokens exceed the limit, the oldest content drops out of the model's view.

What competes for context space

Every API call draws from the same pool of tokens:

	Typical size	Notes
System prompt	100–2,000 tok	Developer-set; charged on every call
Conversation history	Grows with each turn	Main driver of overflow in long sessions
Current user message	1–500 tok typical	Pasted documents can reach 10,000+
Response budget	Controlled by max_tokens	Counts against the same window limit

Token consumers in a single API call

Show typical context window sizes across current models

Window sizes have grown rapidly since GPT-3:

Model family	Approximate context
GPT-3.5 (original)	4,096 tokens
GPT-4 Turbo	128,000 tokens
Claude 3.5 Sonnet	200,000 tokens
Gemini 1.5 Pro	1,000,000 tokens

^{[Context window sizes per vendor documentation (OpenAI, Anthropic, Google)]}

A larger window delays the hard cutoff — it does not eliminate it. Research also shows that models use information near the start and end of a very long context better than information buried in the middle: the “lost in the middle” effect. ^{[Lost in the Middle: How Language Models Use Long Contexts]}

Check your understanding

A user has a 60-turn conversation. Late in the session the model seems to 'forget' the user's name from message 1. What most likely happened?

Where the model breaks — and why that's predictable

Armed with the three-part model — tokens, next-token prediction, context window — the failure modes stop feeling random and start feeling like natural consequences:

	Symptom	Root cause	Practitioner fix
Hallucinated facts	Confident wrong answers	Pattern completion with no truth-verification step	Inject retrieved facts into context (RAG)
Forgets earlier context	Contradicts turn-1 instructions	Context window eviction	Keep critical instructions near the end; re-inject on overflow
Letter-counting errors	Miscounts letters in words	Individual letters invisible below token level	Space-separate letters: 's t r a w b e r r y'
Non-deterministic outputs	Same prompt, different answers	Stochastic sampling at temperature > 0	Set temperature = 0 for reproducible factual tasks

Common LLM failure modes decoded through the three-mechanism model

Show the formal layer: what the attention mechanism actually computes

A transformer processes the full token sequence in parallel. Each token attends to every other token in the context via the self-attention mechanism:

$\text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$

The matrices $Q$ (query), $K$ (key), $V$ (value) are learned during training. Each attention head asks: “which other tokens in the window are most relevant to what comes after me?” Stacked 32–120 layers deep, this process produces final token logits. The crucial point: a token at position 500 can attend directly to position 3 — but only if both sit inside the current context window. ^{[Attention Is All You Need]}

Check your understanding

An LLM gives a plausible-sounding but wrong year for a historical event. Which mechanism best explains this?

Check your mental modelQ 1 / 5

Which of the following is NOT a direct consequence of next-token prediction?