PyTorch 101

Creating Tensors

import torch

# From a Python list
x = torch.tensor([1, 2, 3])         # 1-D, shape (3,)
x = torch.tensor([[1, 2], [3, 4]])  # 2-D, shape (2, 2)

# Special constructors
torch.zeros(2, 3)                    # 2x3 of zeros
torch.ones(5)                        # length-5 of ones
torch.arange(10)                     # [0,1,…,9]
torch.randn(4, 8)                    # Gaussian noise, shape (4,8)
torch.randint(0, 100, (2, 5))        # random ints in [0,100)

Inspecting Tensors

x.shape         # torch.Size([2, 2])
x.dtype         # torch.int64  (int literals; 1.0 → float32)
x.device        # device(type='cpu')
x.numel()       # 4 — total number of elements

Visualizing Shape

A tensor of shape (2, 3):

A 4-D tensor in deep learning is typically (batch, time, heads, dim). You'll see this constantly in attention.

A Tiny Example

# Two learnable scalars
w = torch.tensor([2.0], requires_grad=True)
b = torch.tensor([1.0], requires_grad=True)
x = torch.tensor([3.0])              # input (no grad)

# Forward pass — every op gets recorded
a = w * x         # 6.0   grad_fn=<MulBackward0>
z = a + b         # 7.0   grad_fn=<AddBackward0>
y = z ** 2        # 49.0  grad_fn=<PowBackward0>

print(y.grad_fn)
# <PowBackward0 object at 0x…>

Autograd stores local derivative rules and composes them by the chain rule. It does not invert or undo the forward operation; noninvertible functions can still have useful derivatives.

Three Kinds of Tensors in the Graph

Orange-rimmed = leaf with requires_grad. Purple = op node (knows how to compute its derivative). Green = output we'll backprop from.

Local Derivative for Each Op

Each op type knows its own forward formula and its own derivative. PyTorch ships these as built-in Backward classes.

Chain Them at Each Leaf

# Seed: ∂y/∂y = 1

∂y/∂w = ∂y/∂z · ∂z/∂a · ∂a/∂w
      =   14   ·   1   ·   3     = 42

∂y/∂b = ∂y/∂z · ∂z/∂b
      =   14   ·   1            = 14

PyTorch does exactly this — but for millions of parameters at once, with matrix-valued local derivatives (Jacobians).

The Code That Triggers All This

w = torch.tensor([2.0], requires_grad=True)
b = torch.tensor([1.0], requires_grad=True)
x = torch.tensor([3.0])

y = (w * x + b) ** 2      # 49.0
y.backward()                # ← runs the whole reverse pass

print(w.grad)              # tensor([42.])
print(b.grad)              # tensor([14.])

The Code

import torch

x = torch.tensor([1., 2., 3., 4., 5.])
y = 2 * x + 1                       # the "true" rule

w = torch.zeros(1, requires_grad=True)  # start knowing nothing
b = torch.zeros(1, requires_grad=True)
lr = 0.05

for step in range(501):
    pred = w * x + b                  # 1. forward
    loss = ((pred - y) ** 2).mean()    # 2. MSE loss
    loss.backward()                   # 3. fills w.grad, b.grad
    if step in (0, 1, 10, 100, 500):
        print(step, loss.item(), w.item(), b.item())
    with torch.no_grad():             # 4. update, not tracked
        w -= lr * w.grad
        b -= lr * b.grad
    w.grad.zero_(); b.grad.zero_()      # 5. clear for next step

What It Prints

step   0  loss  57.0000  w 0.000  b 0.000
step   1  loss   1.9400  w 2.500  b 0.700
step  10  loss   0.0222  w 2.097  b 0.651
step 100  loss   0.0010  w 2.021  b 0.925
step 500  loss   0.0000  w 2.000  b 1.000

It finds w = 2, b = 1. Nobody told it the rule. The gradients pointed the way.

Check Step 0 By Hand

• All preds are 0, errors are −3, −5, −7, −9, −11. Mean of squares = 285 / 5 = 57.
• dL/dw = (2/5)·Σ(pred−y)·x = (2/5)·(−125) = −50
• dL/db = (2/5)·Σ(pred−y) = (2/5)·(−35) = −14
• Update: w = 0 − 0.05·(−50) = 2.5, b = 0 − 0.05·(−14) = 0.7. Matches step 1.

The Pattern

import torch.nn as nn
import torch.nn.functional as F

class FeedForward(nn.Module):
    def __init__(self, d_model):
        super().__init__()
        # declare learnable sub-modules here
        self.fc1 = nn.Linear(d_model, 4 * d_model)
        self.fc2 = nn.Linear(4 * d_model, d_model)

    def forward(self, x):
        # describe the computation
        return self.fc2(F.gelu(self.fc1(x)))

# use it
ff = FeedForward(128)
y  = ff(x)        # calls forward() under the hood

What Module Does For You

• Auto-registers sub-modules and parameters — anything assigned to self.X that's a Module or Parameter.
• model.parameters() returns an iterator over all learnable tensors recursively. The optimizer uses this.
• model.to(device) moves everything at once.
• model.state_dict() dumps all weights for saving.
• model.train() / model.eval() toggles modes (dropout/batchnorm care).

Side by Side

Two ways to apply GeLU. Both produce the same output:

class FFN(nn.Module):
    def __init__(self):
        super().__init__()
        self.act = nn.GELU()
        self.fc  = nn.Linear(128, 128)

    def forward(self, x):
        return self.fc(self.act(x))

class FFN(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Linear(128, 128)

    def forward(self, x):
        return self.fc(F.gelu(x))   # ← cleaner

The Functional Ops You'll Use

Indexing & Masking

x = torch.arange(24).view(2, 3, 4)   # shape (2, 3, 4)

x[0]            # first batch:           shape (3, 4)
x[:, -1, :]     # last token, all batch: shape (2, 4)
x[..., 0]       # first elem of last dim: shape (2, 3)
x[:, :, :2]     # first 2 along last dim: shape (2, 3, 2)

x = torch.tensor([10, 20, 30, 40, 50])
idx = torch.tensor([0, 2, 4])

x[idx]
# tensor([10, 30, 50])

# This is exactly how nn.Embedding works internally

x = torch.tensor([1., 2., 3., 4.])
mask = torch.tensor([True, False, True, False])

x.masked_fill(mask, float('-inf'))
# tensor([-inf, 2., -inf, 4.])

# Causal mask in attention:
mask = torch.triu(torch.ones(T,T), diagonal=1).bool()
scores.masked_fill(mask, float('-inf'))   # future → -inf → 0 after softmax

Sampling & Random

torch.randint(0, 100, (2, 3))
# tensor([[42,  7, 91],
#         [ 3, 55, 28]])   shape (2, 3)  — used in get_batch()

torch.randn(3)
# tensor([0.21, -1.45, 0.83])  — mean 0, std 1 (used to init weights)

probs = torch.tensor([0.1, 0.7, 0.2])
torch.multinomial(probs, num_samples=1)
# tensor([1])    — most likely 1 (highest prob), occasionally 0 or 2
# Used in generate() to sample the next token

torch.arange(4)               # tensor([0, 1, 2, 3])  — position indices

torch.tril(torch.ones(4, 4))
# tensor([[1, 0, 0, 0],
#         [1, 1, 0, 0],
#         [1, 1, 1, 0],
#         [1, 1, 1, 1]])    ← causal mask in attention

Worked Examples

x    = torch.randn(32, 128)        # (B, D)
bias = torch.randn(128)             # (D,)

x + bias
# bias is treated as (1, 128) → expanded to (32, 128)
# Same vector added to every row

col = torch.tensor([[1], [2], [3]])    # shape (3, 1)
row = torch.tensor([[10, 20, 30, 40]]) # shape (1, 4)

col + row
# tensor([[11, 21, 31, 41],
#         [12, 22, 32, 42],
#         [13, 23, 33, 43]])    shape (3, 4)

tok = self.tok_embed(idx)        # (B, T, D)
pos = self.pos_embed(torch.arange(T))   # (T, D)

x = tok + pos
# pos is treated as (1, T, D) → expanded to (B, T, D)
# Same position vectors added to every batch item

a = torch.randn(3, 4)
b = torch.randn(3, 5)
a + b   # RuntimeError: 4 ≠ 5 and neither is 1

One Forward Pass, Line by Line

Rules of Thumb

• Linear layers only touch the last dim. (B,T,64) → (B,T,192).
• @ multiplies the last two dims and broadcasts the rest.
• transpose makes a tensor non-contiguous. Call .contiguous() before .view(), or use .reshape().
• cross_entropy wants classes on dim 1. Flatten (B,T,V) to (B·T,V).

Common Optimizers — When to Use What

Quick Code

torch.optim.SGD(model.parameters(), lr=0.05, momentum=0.9)
torch.optim.Adam(model.parameters(), lr=3e-4)
torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.1)
                                          # LLM default

Rules of thumb: training a transformer → AdamW · training a CNN → SGD+momentum · unsure → AdamW with lr=3e-4 ("the Karpathy constant") · out of GPU memory → switch to Adafactor or 8-bit Adam (bitsandbytes).

Pair any of these with a learning-rate scheduler (cosine decay, warmup) — the schedule often matters more than the optimizer choice.

Order Matters — Why?

Mnemonic: Z·F·B·S — Zero, Forward, Backward, Step.

Bonus: Gradient Accumulation

Want a bigger effective batch than fits in memory? Skip zero_grad and step for K steps:

for i, (x, y) in enumerate(dataloader):
    loss = model(x, y) / ACCUM_STEPS
    loss.backward()                # grads keep accumulating

    if (i + 1) % ACCUM_STEPS == 0:   # every K steps:
        optimizer.step()             #   apply combined grads
        optimizer.zero_grad()        #   then clear

Gradient accumulation matches a larger batch only with correct per-example loss weighting and compatible model behavior. BatchNorm statistics, dropout randomness, and a partial final accumulation group can change equivalence.

A 3-Step Sanity Check

• Overfit one batch. Loss should go near 0. If not, the bug is in the model or loop, not the data.
• Check the first loss. With V classes and random init it should be about ln(V). For V=65, about 4.17.
• Print shapes once at the top of forward().

Two Things to Toggle

# As a context manager
with torch.no_grad():
    logits = model(x)
    preds  = logits.argmax(dim=-1)

# Or as a decorator (cleaner for whole functions)
@torch.no_grad()
def generate(model, idx, max_new_tokens):
    for _ in range(max_new_tokens):
        ...
        idx = torch.cat([idx, next_token], dim=1)
    return idx

model.eval()      # dropout off, batchnorm uses running stats
# …generate / predict…
model.train()     # back to training mode

Why Bother?

• Memory: autograd stores intermediate activations for the backward pass. no_grad skips this — often a big memory saving (how much depends on the model).
• Speed: no graph construction, no .grad bookkeeping → usually noticeably faster.
• Correctness: Dropout layers behave wildly differently in train vs eval mode. Forgetting model.eval() is a classic bug.

The Three-Step Inference Recipe

model.eval()
with torch.no_grad():
    output = model(x)

Run this anytime you're predicting/generating, not training.

Saving

# Get a dict of {param_name → tensor}
state = model.state_dict()

# Save it (plus anything else you'll need later)
torch.save({
    "model":     state,
    "optimizer": optimizer.state_dict(),
    "config":    config,
    "step":      step,
}, "checkpoints/model.pt")

Loading

ckpt = torch.load(
    "checkpoints/model.pt",
    map_location=device,        # load onto current device
    weights_only=False,        # allow non-tensor objects (config)
)                               # ⚠ False runs pickle: ONLY for files you trust.
                                #   Default is True since PyTorch 2.6.

# Rebuild the architecture from code
model = GPT(ckpt["config"]).to(device)
model.load_state_dict(ckpt["model"])

What's a state_dict?

An ordered dict mapping every parameter (and buffer) to its tensor. For our tiny GPT it looks roughly like:

{
  "tok_embed.weight":        tensor(50257, 128),
  "pos_embed.weight":        tensor(128, 128),
  "blocks.0.ln1.weight":     tensor(128),
  "blocks.0.ln1.bias":       tensor(128),
  "blocks.0.attn.qkv.weight": tensor(384, 128),
  "blocks.0.attn.proj.weight": tensor(128, 128),
  "blocks.0.ffn.fc1.weight":  tensor(512, 128),
  "blocks.0.ffn.fc1.bias":    tensor(512),
  ...
}

Picking the Best Available Device

device = (
    "cuda"  if torch.cuda.is_available()
    else "mps"   if torch.backends.mps.is_available()
    else "cpu"
)
print(f"running on {device}")

Moving Things Around

# Move the whole model
model = model.to(device)

# Move tensors
x = x.to(device)
y = y.to(device)

# Or create directly on a device
x = torch.zeros(3, 4, device=device)

Speed Comparison (rough)

For training large models, GPUs aren't a luxury — they're a necessity.

Bonus: map_location

ckpt = torch.load("model.pt", map_location=device)

Load a checkpoint that was saved on a CUDA machine onto a CPU/MPS machine. Without this you'll hit a "no CUDA" error.

What is GGUF?

GGUF = GPT-Generated Unified Format. A binary container designed for fast, memory-mapped inference. Built by Georgi Gerganov for llama.cpp; now the de-facto standard for local LLM deployment.

What's Inside

• Model weights — usually quantized to 2–8 bits
• Tokenizer — vocab + merges, embedded in the file
• Architecture metadata — layer count, head count, context length
• Chat template — how to wrap conversations

GGUF bundles model data and metadata, but it still needs a compatible inference runtime. Memory mapping defers I/O and does not make a large model instantly resident or free to execute.

Quantization Levels — Pick Your Trade-off

A 7B Llama at Q4_K_M weighs ~4 GB and runs comfortably on a 16 GB laptop.

The Ollama Modelfile

Like a Dockerfile for AI models. Declares: base weights, system prompt, sampling defaults, chat template. Build it once → run it anywhere Ollama runs.

Basic Modelfile Syntax

FROM      <base>           # required: model to start from
SYSTEM    "..."            # default system prompt
PARAMETER temperature 0.7  # sampling knobs
PARAMETER top_p       0.9
PARAMETER num_ctx     8192
TEMPLATE  "..."            # optional: chat format override
LICENSE   "..."            # optional: license text

Example: A Custom Python Tutor

# Modelfile
FROM llama3.2:3b

SYSTEM """You are a patient Python tutor.
Always show runnable code examples and
explain edge cases."""

PARAMETER temperature 0.5
PARAMETER top_p       0.9
PARAMETER num_ctx     8192
PARAMETER stop        "<|eot_id|>"

Build & Run

$ ollama create py-tutor -f Modelfile
# transferring model data 100%
# creating model layer
# writing manifest
# success

$ ollama list
# NAME              ID            SIZE     MODIFIED
# py-tutor:latest   a3b1c…        2.0 GB   3 minutes ago
# llama3.2:3b       7d8c9…        2.0 GB   2 days ago

$ ollama show py-tutor              # config, params, template
$ ollama show py-tutor --modelfile  # dump the Modelfile
$ ollama cp py-tutor py-tutor-v2    # clone
$ ollama rm py-tutor                # delete
$ ollama pull llama3.2:3b           # fetch from registry
$ ollama push user/py-tutor         # publish to ollama.com

$ ollama run py-tutor
>>> Explain decorators with an example.
# …streaming response…
>>> /set system "Now answer in haikus."   # override at runtime
>>> /show modelfile                          # inspect
>>> /bye                                     # exit

$ ollama run py-tutor "Why is Python slow?"

$ echo "Summarize this:" | ollama run py-tutor
cat log.txt | ollama run py-tutor "Find errors:"

$ ollama serve              # starts on http://localhost:11434

# Generate (single-turn)
$ curl http://localhost:11434/api/generate -d '{
    "model": "py-tutor",
    "prompt": "Explain async/await",
    "stream": false
  }'

# Chat (multi-turn, OpenAI-style messages)
$ curl http://localhost:11434/api/chat -d '{
    "model": "py-tutor",
    "messages": [
      {"role": "user", "content": "Hi"}
    ]
  }'

# pip install ollama
from ollama import chat

response = chat(
    model="py-tutor",
    messages=[{"role": "user", "content": "Explain GIL"}],
)
print(response.message.content)

# Or stream tokens as they arrive
for chunk in chat(model="py-tutor", messages=[...], stream=True):
    print(chunk.message.content, end="", flush=True)

Loading Your Own GGUF

# Modelfile
FROM ./my-model.Q4_K_M.gguf
SYSTEM "You are my fine-tuned model."
PARAMETER temperature 0.3

FROM can reference a compatible local GGUF model supported by the installed runtime. Arbitrary architectures or incompatible tokenizer metadata are not automatically supported.

Autograd

Layers

Shapes

Training & Inference

The Core Concepts

What to Read Next

• Official tutorials: pytorch.org/tutorials — the "Learn the Basics" track
• Karpathy's nanoGPT: the canonical small GPT in PyTorch
• The PyTorch source: reading nn/modules/linear.py teaches you how to write your own layer
• The complete self-contained line-fitting program and tests are embedded in this deck and available under examples/. No external tiny_llm project is required.

You Now Have Enough To…

• Read and modify the complete included PyTorch example, then extend its data, model, loss, or evaluation.
• ✓ Modify the architecture (add layers, change dimensions)
• ✓ Write your own training loop on a different dataset
• ✓ Save and resume training
• ✓ Move work to / from GPU
• ✓ Debug shape and device errors