How Large Language Models Work

The Job of a Tokenizer

• Convert raw text into a sequence of integer IDs the model can ingest.
• Reverse the process: turn the model's predicted IDs back into text (detokenization).
• Maintain a fixed vocabulary — every ID maps to one token.

Three Approaches

• Character-level — tiny vocab (~100), but very long sequences.
• Word-level — short sequences, but huge vocab and tons of "unknown" words.
• Subword (BPE, WordPiece, SentencePiece) — best of both. Common words stay whole, rare words split into pieces. This is what modern LLMs use.

What Tokenization Looks Like

"The cat sat on the mat."

[791, 8415, 7731, 389, 279, 5634, 13]

The model never sees "cat" — it sees the integer 8415. The output will be another integer, which gets detokenized back to text.

"unbelievable"

u n b e l i e v a b l e

[403, 8265, 480]

IDs are illustrative — real splits vary by tokenizer.

"The cat sat on the mat."

Note the leading spaces — whitespace is part of the token.

What Are Special Tokens?

• Reserved IDs in the vocabulary that don't appear in normal text.
• Added after BPE training — typically a few hundred slots (e.g. IDs 128000–128255 in Llama 3).
• To the model they're just integers — but it's fine-tuned to give them specific meanings.
• Special-token handling depends on tokenizer and API serialization policy. Trusted role boundaries must be constructed by the runtime; token reservation alone does not prevent prompt injection.

Common Categories

Chat Template — A Real Example

messages = [
  {"role": "system",    "content": "You are helpful."},
  {"role": "user",      "content": "Hi!"},
  {"role": "assistant", "content": "Hello!"}
]

<|begin_of_text|>
<|start_header_id|>system<|end_header_id|>
You are helpful.<|eot_id|>
<|start_header_id|>user<|end_header_id|>
Hi!<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>
Hello!<|eot_id|>

[128000, 128006, 9125, 128007, 271, 2675, 527, 11190, 13,
 128009, 128006, 882, 128007, 271, 13347, 0, 128009,
 128006, 78191, 128007, 271, 9906, 0, 128009]

IDs ≥ 128000 are special. 271 is the "\n\n" that follows every header. The model learns: "after assistant<|end_header_id|>, generate a reply, then emit <|eot_id|> to stop."

The Embedding Table

A matrix E ∈ ℝ^{V × d} where:

• V = vocabulary size (e.g. 128,000)
• d = model dimension (e.g. 4,096) — the length of every vector inside the model

token_id  →  E[token_id]  ∈  ℝ⁴⁰⁹⁶

  403  →  [ 0.12, -0.84, 0.31, …, 0.07]
 8265  →  [-0.45,  0.22, 0.91, …,-0.18]
  480  →  [ 0.67, -0.13, 0.04, …, 0.55]

Plus a positional encoding is added so the model knows token order.

Vector Arithmetic Works

vec("king") − vec("man") + vec("woman") ≈ vec("queen")
vec("Paris") − vec("France") + vec("Italy") ≈ vec("Rome")

Direction in the space encodes meaning. Distance encodes similarity.

The Problem

Attention weights only depend on token content, not order. So we have to add the order back in explicitly.

x_i = embedding(token_i) + position_vector(i)
       └─ what it is ─┘   └─── where it sits ───┘

Same shape as the embedding (ℝᵈ), so addition just works. The transformer now receives both signals fused into one vector.

Three Flavors

Fixed sin/cos at exponentially varying frequencies. No parameters. Defined for any length — though quality still drops past the lengths seen in training.

A second embedding table indexed by position (0…N−1). Trained via backprop. Cannot extrapolate past max length.

Rotary Position Embedding — rotates Q and K vectors by an angle proportional to position. Encodes relative distance directly. Extends to longer contexts with scaling tricks (position interpolation, NTK / YaRN).

Learned Encoding — A Second Embedding Table

Treat each position like a token. Allocate a second lookup table P ∈ ℝ^{N_max × d} and train it via backprop alongside everything else.

position_id  →  P[position_id]  ∈  ℝᵈ

  0  →  [ 0.21, -0.74, 0.55, …, -0.08]   ← learned
  1  →  [-0.13,  0.62, 0.41, …,  0.27]   ← learned
  2  →  [ 0.08, -0.31, 0.92, …, -0.44]   ← learned
  …
N_max−1 →  [ …                         ]

Each input vector becomes the sum of two learned lookups:

The Three Roles

From each token's embedding x, three vectors are projected via learned weight matrices:

Q = x · W_Q   ← "what am I looking for?"
K = x · W_K   ← "what do I offer?"
V = x · W_V   ← "what info do I carry?"

The Attention Formula

Attention(Q, K, V) = softmax( Q·Kᵀ / √d ) · V

• Q · Kᵀ — dot product = compatibility score for every token pair
• / √d — scale to keep gradients stable (d = d_k, per-head size)
• softmax — turn scores into a probability distribution (the "attention weights")
• · V — weighted sum of values = context-aware new representation

Sentence: "The cat sat on the mat"

When processing "sat", it can only attend to itself and prior tokens.

new_representation = 0.10·V_The + 0.75·V_cat + 0.15·V_sat
                   (future V_on, V_the, V_mat are excluded)

Setup (W_K = W_V = I, W_Q doubles dim 2)

Result for Every Row (causal)

Each row sums to 1. "The" can only see itself. "sat" pulls most from itself and "cat" — its best-matching keys.

The "sat" Row, Step by Step

1. scores   q·k   = [1·1+2·0, 1·0+2·1, 1·1+2·1]
                  = [1, 2, 3]
2. scale    ÷ √2  = [0.71, 1.41, 2.12]
3. softmax        = [0.14, 0.28, 0.58]
4. blend V  0.14·[1,0] + 0.28·[0,1] + 0.58·[1,1]
                  = [0.72, 0.86]

Run It Yourself (plain Python)

import math
Q = [[1, 0], [0, 2], [1, 2]]          # The, cat, sat
K = V = [[1, 0], [0, 1], [1, 1]]

def softmax(v):
    m = max(v); e = [math.exp(x - m) for x in v]
    return [x / sum(e) for x in e]

for t in range(3):                     # query position
    s = [sum(q * k for q, k in zip(Q[t], K[j])) / math.sqrt(2)
         for j in range(t + 1)]        # causal: keys 0..t only
    w = softmax(s)
    out = [sum(w[j] * V[j][c] for j in range(t + 1)) for c in range(2)]
    print(t, [round(x, 2) for x in w], [round(x, 2) for x in out])

The Math: Variance Grows With d

Assume each component of Q and K is roughly independent with mean 0 and variance 1. Then for a single dot product:

q · k = Σᵢ qᵢ · kᵢ      (i = 1 … d)

E[q · k]    = 0
Var[q · k]  = d
Std[q · k]  = √d

So as d grows (typical: 64, 128, even 256 per head), the raw scores q·k grow proportionally to √d.

For d=128 → typical scores ≈ ±11 (so 22 is just a 2σ score). For d=4096 → typical scores ≈ ±64.

The Fix

softmax( Q·Kᵀ / √d ) · V

• Dividing by √d rescales the variance back to 1, regardless of head size.
• Softmax now operates in its sensitive range — outputs are spread across tokens, not collapsed to one.
• Gradients flow → the model can learn attention patterns.
• Trains stably across model sizes — same formula whether d=64 or d=512.

The Failure Mode: Softmax Saturation

Softmax is exponential. When inputs are large, one value dominates:

scores  = [22, 8, 14, 17]
softmax = [0.99, 0.00, 0.00, 0.01]

scores  = [1.94, 0.71, 1.24, 1.50]
softmax = [0.41, 0.12, 0.20, 0.27]

Why It's Required

• Training objective: predict token t+1 from tokens 1…t. If token t could see t+1, the model would just copy the answer — learning nothing.
• Inference consistency: at generation time, future tokens literally don't exist yet. The model must work with only the prefix.
• Parallel training: the mask lets us compute predictions for all positions at once in a single forward pass — instead of running the model N times.

How It's Implemented

After computing Q·Kᵀ / √d but before softmax, add a mask matrix M:

scores = Q·Kᵀ / √d
scores = scores + M           ← M is upper-triangular −∞
attn   = softmax(scores)      ← e^−∞ = 0, so future is gone
output = attn · V

Setting masked entries to −∞ (not 0) means after softmax they become exactly 0 — and the visible probabilities still sum to 1.

Each row = one query token. Each column = a key it might attend to. Green = allowed, red = masked (−∞).

GPT-2 Small: V = 50,257 · d = 768 · L = 12 · N_max = 1,024

That's the famous "124M". Rule of thumb for the blocks alone: 12·L·d² = 12·12·768² ≈ 85M.

Compute Rules of Thumb (N = parameter count)

• Generating one token ≈ 2N FLOPs (one multiply + one add per weight).
• Training on one token ≈ 6N FLOPs (forward 2N + backward ≈ 4N).
• So a 7B model needs ≈ 14 GFLOPs per generated token (ignoring attention over the context).

Multi-Turn Conversation

A stateless transformer function consumes a supplied context. The application or serving runtime may persist conversation state, summaries, or reusable KV prefixes across turns.

turn 1 prompt:
  <sys>… <user>Hi</> <asst>

turn 2 prompt:
  <sys>… <user>Hi</>
  <asst>Hello!</>
  <user>How are you?</> <asst>

turn 3 prompt:
  <sys>… <user>Hi</>
  <asst>Hello!</>
  <user>How are you?</>
  <asst>Doing well!</>
  <user>What did I say first?</> <asst>

This is why long conversations get expensive — the prompt grows unboundedly until it hits the context window limit.

Sampling Knobs You Control

• temperature — 0 = deterministic; 1 = balanced; >1 = wild
• top-p / top-k — restrict the candidate pool before sampling
• max_tokens — hard cap on response length
• stop sequences — additional strings that halt generation

How Tools Change the Prompt

Tools are added declaratively in the system prompt as JSON schemas. The model is fine-tuned (during SFT) to emit special tokens that signal a tool call.

<|sys|>You are a helpful assistant.<|/>
<|user|>What's the weather in SF?<|/>
<|asst|>

<|sys|>You are a helpful assistant.

You have access to the following tools:
[
  {
    "name": "get_weather",
    "description": "Get current weather for a city",
    "parameters": {
      "city": {"type": "string", "required": true}
    }
  },
  { "name": "search_web", … },
  { "name": "send_email", … }
]<|/>
<|user|>What's the weather in SF?<|/>
<|asst|>

<|tool_call|>
{"name": "get_weather", "args": {"city": "SF"}}
<|/tool_call|>

<|tool_result|>
{"temp_f": 62, "conditions": "foggy"}
<|/tool_result|>
<|asst|>It's 62°F and foggy in SF right now.<|/>

The model never actually called an API — it just emitted text. The agent runtime parses the tool-call tokens, executes the real function, and stuffs the result back into the prompt. Same loop the chatbot uses, just with extra structured turns.

How Memory Is Added

The model itself is stateless — it forgets everything between API calls. "Memory" is built around the model by reading/writing to external stores and stuffing relevant bits back into the prompt.

<|sys|>You are a helpful assistant.

## Recalled memory (relevant facts):
- User prefers metric units
- User is based in Berlin
- Last conversation: discussed Python tooling

[tool definitions…]<|/>
<|user|>What's the weather?<|/>

Many transformer applications serialize retrieved memory into context text. Other designs use structured inputs, cached vectors, multimodal representations, or recurrent state; memory is not universally extra text.

The Five Layers We Covered

1. Architecture — input → tokenizer → embeddings → N transformer blocks → LM head → output
2. Tokenization — byte-level BPE turns any text into a sequence of integer IDs
3. Embeddings — IDs index into a learned table of vectors; meaning lives in geometry
4. Attention — Q·Kᵀ scores, softmax weights, weighted sum of V; multi-head = parallel perspectives
5. Output — final hidden state → logits → softmax → sampled token; repeat

Where to Go Next

• Attention Is All You Need (Vaswani et al., 2017) — the original paper
• The Illustrated Transformer — Jay Alammar's visual guide
• nanoGPT by Karpathy — a full LLM in ~300 lines of code
• 3Blue1Brown — But what is a GPT? — best video explanation