Physics-Informed Neural Networks
Learn the ideas from the beginning, then train a small PyTorch model that predicts how a warm rod cools.
↓ neural network
predicted temperature û
↓ checked against physics
Four parts, in a useful order
1 · Learn what a network does
Functions, neurons, predictions, loss, and training.
2 · Read the physics
Change, slopes, curvature, the heat equation, and conditions.
3 · Build a PINN
Residuals, sampling, loss terms, and automatic differentiation.
4 · Run and check it
Tensors, the PyTorch code, training results, and experiments.
What are we trying to predict?
A physical system
A thin rod conducts heat along its length.
A question
How warm is the rod at a particular position and time?
output: temperature
One prediction needs two coordinates
t = 0.50 → halfway through time
u(0.25, 0.50) = temperature
Read the notation
x tells us where we are. t tells us when.
u is the name we give the temperature field: a value for every (x,t).
A function turns an input into an output
f(0) = 1
f(2) = 5
f(3) = 7
Evaluate a function
Choose an input. Substitute it into the rule. Calculate the output.
Our final rule needs two inputs: u(x,t).
A data point is one observed example
| x | t | u |
|---|---|---|
| 0.50 | 0.50 | 0.610498 |
What the row says
At the middle of the rod, halfway through our time interval, the temperature is about 0.61.
We can compare a prediction at (0.50,0.50) with this number.
Learning means adjusting a rule
w = 1, b = 0, x = 2
ŷ = 2
observed answer: y = 5
Parameters
w is a weight: it scales the input.
b is a bias: it adds an offset.
Training changes w and b to bring predictions closer to the examples.
A neuron combines inputs, then transforms them
a = tanh(z)
x=.5, t=.2, w₁=2, w₂=−1, b=0
z = 1 − .2 = .8
a ≈ .664
Two operations
1. Compute a weighted sum z.
2. Apply an activation function to get a.
The activation lets a network represent curved, more complex functions.
Try changing a neuron's weight
Hold x = 0.5, t = 0.2, w₂ = −1, b = 0
tanh(z) = 0.664
Watch the effect
Moving the weight changes the weighted sum and the neuron's output.
Training does this automatically for every weight in the network.
Why do we use tanh?
tanh(0) = 0
tanh(2) ≈ 0.964
A smooth activation
tanh bends smoothly between −1 and +1.
Without activations, a stack of linear layers is still just a linear function.
Smoothness will matter when we take second derivatives.
A layer applies many neurons at once
x, t
tanh
tanh
tanh
û
Each layer uses the previous layer's outputs as its inputs. This is a multilayer perceptron, or MLP.
A forward pass produces a prediction
↓ hidden layers
û = 0.48
Before training
We initialize weights to small random values.
The first prediction will generally be inaccurate.
A forward pass evaluates the current network; it does not change any weights.
A loss measures how wrong a prediction is
0.48 − 0.61 = −0.13
squared error = (−0.13)²
= 0.0169
Why square the error?
Positive and negative mistakes cannot cancel.
Larger mistakes receive a larger penalty.
A perfect prediction has squared error zero.
Mean squared error averages many mistakes
| error | error² |
|---|---|
| −0.1 | 0.01 |
| +0.2 | 0.04 |
| 0.0 | 0.00 |
= 0.016667
One weight update, calculated by hand
L = (2w − 5)² = 9
dL/dw = 4(2w − 5) = −12
w_new = 1 − 0.1(−12) = 2.2
new L = (4.4 − 5)² = 0.36
Gradient descent
A gradient tells us how loss changes when a weight changes.
Move the weight opposite the gradient.
The learning rate, here 0.1, sets the step size.
Training repeats the same four operations
- Predict outputs with the current weights.
- Calculate the loss from those predictions.
- Backpropagate: compute the loss gradient for every weight.
- Use an optimizer to update the weights.
What do measurements leave unknown?
Sparse observations
A few sensors tell us the temperature at a few coordinates.
Many functions can match those observed values.
Known physical law
The heat equation constrains how temperature can change throughout the rod.
PINNs use that law to supply extra training information.
A rate is change divided by an interval
time: 0.20 → 0.30
average rate = −0.10 / 0.10
= −1.0
Read the sign
A negative rate means the temperature is decreasing.
The rate is an average over that time interval.
A derivative is the slope at one point
near x=2:
(2.1² − 2²) / 0.1 = 4.1
(2.01² − 2²) / 0.01 = 4.01
f′(2) = 4
Shrink the step
The closer the two x values are, the closer this estimate gets to the local slope.
For x², the exact derivative is f′(x)=2x.
With two inputs, change one at a time
∂u/∂x = 2x (hold t fixed)
∂u/∂t = 3 (hold x fixed)
Partial derivative
The symbol ∂ tells us the function has several inputs.
A partial derivative changes one input while holding the others fixed.
A second derivative measures changing slope
f′(x) = 2x
f″(x) = 2
g(x) = 1 − x²
g″(x) = −2
Curvature
Positive second derivative: the curve bends upward.
Negative second derivative: the curve bends downward.
Zero second derivative: a locally straight profile.
Heat exchange explains curvature
0.6 + 0.6 − 2(1.0) = −0.8
The center cools
The center is hotter than both neighbors, so heat flows outward.
The negative difference corresponds to negative spatial curvature.
The heat equation connects time and curvature
u_t = αu_xx
Read the law
The rate of temperature change equals diffusivity times spatial curvature.
If curvature is negative, temperature decreases; if positive, it increases.
Diffusivity α controls how quickly heat spreads
α = 0.1 → u_t = −0.4
α = 0.01 → u_t = −0.04
A physical coefficient
Larger α means faster smoothing.
In our example α=0.1 is known and fixed.
The neural-network weights θ are different: they represent the solution.
What makes this a differential equation?
Differential equation
It relates an unknown function to its derivatives.
Solving it means finding a function that satisfies the relation.
PDE = partial differential equation
inputs: x and t
The domain says where the equation applies
0 ≤ t ≤ 1
(0.3,0.8) is inside
(1.4,0.5) is outside
Position × time
Every pair in this rectangle is a place where we want temperature predictions.
We train and evaluate on this domain.
The PDE allows many possible solutions
u(x,t) = 1
for both: u_t = 0, u_xx = 0
both satisfy the heat equation
What is missing?
We must specify the starting temperature and what happens at the ends.
Those conditions identify the physical problem we intend to solve.
The initial condition gives the starting profile
u(0,0) = 0
u(0.5,0) = 1
u(1,0) = 0
At time zero
The ends start cool and the middle starts warm.
We prescribe temperature at every position along the rod.
Boundary conditions describe the ends
u(1,t) = 0
for every time t in [0,1]
Fixed temperatures
Imagine both ends are attached to reservoirs held at the reference temperature.
Heat can leave through those ends.
We use a problem with a known solution
Separate the factors
sin(πx) sets the shape along the rod.
e−απ²t reduces the amplitude over time.
Check that the formula obeys the PDE
u_t = −απ²u
u_xx = −π²u
αu_xx = −απ²u = u_t
Also check conditions
At t=0, the exponential is 1, giving sin(πx).
At x=0 or x=1, the sine is zero.
Calculate one reference temperature
sin(π × .5) = 1
e−.1 × π² × .5 ≈ .610498
u(.5,.5) ≈ .610498
A benchmark
After training, we can query the same coordinate.
The prediction should be close to this value when the model has learned the solution.
Move through time and observe the solution
Both ends stay at zero. The center cools as the curve's amplitude decreases.
What makes a neural network physics-informed?
ordinary training: fit examples
PINN training: also obey the PDE
The same network
The architecture is an ordinary smooth MLP.
The physics enters through the training loss.
We change θ until predictions satisfy the rules well.
A residual measures violation of the PDE
if û_t = −0.30, û_xx = −2
r = −0.30 − .1(−2) = −0.10
Zero is the target
A residual of zero means the prediction satisfies the equation at this coordinate.
Our example residual of −0.10 means it does not.
Turn residuals into a physics loss
Lphysics = mean(r²)
= (0.01 + 0.04 + 0)/3
= 0.016667
Many checks
Evaluate the residual at many position-time pairs.
Square each result and average.
Training pushes that average toward zero.
Physics points need coordinates but no labels
↓ predict û
↓ compute û_t and û_xx
↓ compare residual with zero
Collocation points
This is the name for coordinates used to enforce the differential equation.
We do not need a measured u value at these locations.
Initial-condition loss checks the starting state
at x=.5: target = 1
prediction = .8
squared error = .04
Known targets
Choose positions x and set time to zero.
The prescribed sine profile supplies the correct answer at each position.
Boundary loss keeps both ends at zero
right-end prediction = −.20
Lboundary = (.10² + .20²)/2
= .025
Same rule at many times
Sample times and evaluate both x=0 and x=1.
The target at every one of these points is zero.
Observation loss adds temperature measurements
prediction = .580498
error = −.03
squared error = .0009
Optional evidence
The PDE and conditions already define this forward problem.
Interior observations can help training and are needed in many inverse problems.
Combine all four losses
+ 10 Linitial
+ 10 Lboundary
+ Ldata
Loss weights
The factors 10 give the condition errors more influence at the start.
These are training choices, not constants from the heat equation.
A numerical example of loss weighting
| term | raw loss | weight | contribution |
|---|---|---|---|
| physics | .020 | 1 | .020 |
| initial | .010 | 10 | .100 |
| boundary | .003 | 10 | .030 |
| data | .005 | 1 | .005 |
Total = 0.155
The initial-condition error contributes the most after weighting.
Weighting changes which violations the optimizer prioritizes.
Autograd tracks how outputs depend on inputs
x = torch.tensor([[2.0]],
requires_grad=True)
y = x ** 2
dy_dx = torch.autograd.grad(
y, x, create_graph=True
)[0]
# dy_dx = tensor([[4.]])Automatic differentiation
PyTorch records differentiable operations in a computation graph.
It applies the chain rule backward through those operations to calculate a derivative.
The chain rule handles stacked operations
y = z²
dy/dz = 2z; dz/dx = 2
dy/dx = (2z)(2) = 4z = 8x
at x=1: dy/dx = 8
Why this matters
A neural network stacks many operations.
The derivative of the final output combines the derivatives along its computation paths.
PINNs use two kinds of differentiation
Coordinate derivatives
These build the physics residual.
Parameter gradients
These tell the optimizer how to update the network.
Keep the graph to differentiate twice
u_x = torch.autograd.grad(
u, x, torch.ones_like(u),
create_graph=True
)[0]
u_xx = torch.autograd.grad(
u_x, x, torch.ones_like(u_x),
create_graph=True
)[0]create_graph=True
It keeps first derivatives differentiable, allowing a second derivative.
It also allows the final physics loss to backpropagate to the weights.
Second derivatives favor smooth activations
tanh
smooth second derivative
A practical choice for this heat equation.
ReLU
Its second derivative is zero almost everywhere, and the slope has a kink at zero.
A tensor is an array of numbers
x = torch.tensor([[0.2], [0.5], [0.8]]) t = torch.tensor([[0.1], [0.1], [0.1]]) print(x.shape) # torch.Size([3, 1])
Shape (N,1)
N is the number of examples.
Each row holds one coordinate.
Here we query three positions at the same time.
Combine position and time into each row
coordinates = torch.cat((x, t), dim=1) # [[0.2, 0.1], # [0.5, 0.1], # [0.8, 0.1]] # shape: (3, 2)
dim=1
Concatenate along the column direction.
Each example now has the two inputs the network needs.
Map inputs to a range suitable for tanh
0 → −1
.5 → 0
1 → 1
Input scaling
The domain is [0,1] for both x and t.
This transformation centers inputs near zero, where tanh is sensitive to changes.
nn.Module holds the trainable layers
class PINN(nn.Module):
def __init__(self, width=32):
super().__init__()
self.net = nn.Sequential(
nn.Linear(2, width), nn.Tanh(),
nn.Linear(width, width), nn.Tanh(),
nn.Linear(width, width), nn.Tanh(),
nn.Linear(width, 1),
)Read the architecture
Linear(2,32): two inputs become 32 features.
Two more hidden layers refine those features.
Linear(32,1): predict one temperature.
forward() defines a prediction
def forward(self, x, t):
coordinates = torch.cat((x, t), dim=1)
return self.net(2.0 * coordinates - 1.0)
u = model(x, t)
# shape: (N, 1)A pointwise model
Each row is processed independently.
The output row corresponds to the input (x,t) row.
The CSV provides repeatable observations
x,t,u 0.0,0.0,0.000000000000 0.1,0.0,0.309016994375 0.2,0.0,0.587785252292 ... 0.5,0.5,0.610498025266
121 rows
11 positions × 11 times, including the edges and time zero.
Training selects 16 interior, positive-time rows from this data.
Load and choose observations
x, t, u = load_data(path, device) interior = (x > 0) & (x < 1) & (t > 0) mask = interior.reshape(-1) x, t, u = x[mask], t[mask], u[mask] indices = torch.randperm(len(x))[:16] x_d, t_d, u_d = x[indices], t[indices], u[indices]
Separate kinds of evidence
Initial and boundary values are already enforced by their own loss terms.
The data term uses selected interior observations.
Create physics points with gradient tracking
x_f = torch.rand(1000, 1, device=device,
requires_grad=True)
t_f = torch.rand(1000, 1, device=device,
requires_grad=True)What the flags mean
rand samples uniform coordinates in [0,1).
requires_grad=True lets autograd differentiate outputs with respect to x_f and t_f.
Create initial and boundary points
x_i = torch.linspace(0, 1, 64).reshape(-1, 1) t_i = torch.zeros_like(x_i) u_i = torch.sin(math.pi * x_i) times = torch.linspace(0, 1, 64).reshape(-1, 1) x_b = torch.cat((0 * times, 0 * times + 1)) t_b = torch.cat((times, times))
Fixed conditions
64 positions at t=0 check the initial shape.
64 times on each end give 128 boundary checks.
A small helper calculates derivatives
def derivative(y, x):
return torch.autograd.grad(
y, x,
grad_outputs=torch.ones_like(y),
create_graph=True,
)[0]Read the arguments
y: the outputs to differentiate.
x: the inputs to differentiate against.
ones_like(y): an all-one seed with matching shape.
Implement the differential equation directly
def pde_residual(model, x, t, alpha):
u = model(x, t)
u_t = derivative(u, t)
u_x = derivative(u, x)
u_xx = derivative(u_x, x)
return u_t - alpha * u_xxOne line per mathematical step
Predict temperature.
Compute its time derivative and spatial curvature.
Subtract the two sides of the heat equation.
Implement the four losses
r = pde_residual(model, x_f, t_f, alpha) loss_f = r.square().mean() loss_i = (model(x_i, t_i) - u_i).square().mean() loss_b = model(x_b, t_b).square().mean() loss_d = (model(x_d, t_d) - u_d).square().mean() loss = loss_f + 10*loss_i + 10*loss_b + loss_d
Same pattern
For each rule: calculate a mismatch, square it, average it.
Then combine the scalar terms with weights.
One epoch makes one optimizer update
optimizer.zero_grad(set_to_none=True)
terms = loss_terms(model, physics_points(...),
conditions, data, alpha)
loss = (terms['physics'] + 10*terms['initial']
+ 10*terms['boundary'] + terms['data'])
loss.backward()
optimizer.step()The optimizer
Adam receives model.parameters() and learning rate 0.001.
backward computes weight gradients.
step changes weights; zero_grad clears previous gradients.
Prediction after training is simpler
model.eval()
with torch.no_grad():
x = torch.tensor([[0.5]])
t = torch.tensor([[0.5]])
predicted_u = model(x, t).item()Two different purposes
eval() sets evaluation behavior for certain layers.
no_grad() disables gradient tracking for ordinary predictions.
Residual evaluation still needs gradients.
Save the model so you can use it again
torch.save({
'state_dict': model.state_dict(),
'width': args.width,
'alpha': args.alpha,
}, output / 'pinn_heat_equation.pt')What is stored
state_dict holds learned weights and biases.
width reconstructs the architecture.
alpha records which physical problem was solved.
Download the complete example
Files to save in one folder
What is included
Runnable code with configurable epochs, network width, sampling, and loss weights.
Data loading, plots, metrics, checkpoints, and prediction from saved weights.
Set up Python and install dependencies
python3 -m venv .venv source .venv/bin/activate python -m pip install -r requirements.txt
Why these commands?
venv creates an isolated Python environment.
activate selects it for this terminal.
pip installs PyTorch, NumPy, and Matplotlib.
Run a short test, then train the model
# Quick check: 10 optimizer steps
python pinn_heat_equation.py --epochs 10 \
--n-physics 128 --output-dir output-smoke
# Full tutorial run
python pinn_heat_equation.py --epochs 2500Default configuration
32 neurons per hidden layer.
1,000 fresh physics points per step.
16 interior observations and Adam learning rate 0.001.
Read the training log
epoch 1 | total 7.932e+00
| physics 2.151e-02
| initial 5.412e-01
| boundary 2.189e-01
| data 3.100e-01
epoch 2500 | total 2.006e-04Scientific notation
7.932e+00 means 7.932.
2.006e−04 means 0.0002006.
The total is weighted; component losses are reported before weighting.
Compare the learned field with the exact field
Measure accuracy and physics separately
Relative solution error
‖û−u‖₂ / ‖u‖₂ on a 101×101 reference grid.
Maximum temperature error
Largest absolute difference on that grid.
Physics residual RMSE
sqrt(mean(r²)) at 1,000 newly sampled points.
Loss curves explain how learning progressed
Find the saved model, plots, and numerical logs
| file under output/ | purpose |
|---|---|
| pinn_heat_equation.pt | weights + width + α |
| pinn_results.png | solution and absolute error |
| loss_history.png | training curves |
| loss_history.csv | losses from every step |
| metrics.json | configuration + validation metrics |
Query a trained model without retraining
python pinn_heat_equation.py --predict 0.5 0.5 # For the verified tutorial run: # u(x=0.5, t=0.5) = 0.61132526
trained prediction = 0.61132526
absolute error ≈ 0.000827
Train using physics and conditions only
python pinn_heat_equation.py --data-weight 0 \
--output-dir output-physics-onlyWhat remains?
The PDE residual, initial profile, and fixed end temperatures.
The observation term contributes zero to the total.
Compare the result with the standard run.
Compare with fitting conditions and observations
python pinn_heat_equation.py --physics-weight 0 \
--output-dir output-without-physicsCheck the difference
The network still fits initial values, boundaries, and observations.
The PDE loss no longer influences weight updates.
Compare errors between measurement locations.
Make heat diffuse more slowly
python pinn_heat_equation.py --alpha 0.01 \
--data heat_slow.csv --output-dir output-slowPredict the outcome first
The sine profile should decay more slowly.
At the center and t=1, the exact value becomes about 0.9060.
Reduce the number of physics checks
python pinn_heat_equation.py --n-physics 100 \
--output-dir output-fewer-pointsA useful tradeoff
Each step becomes cheaper.
Each step checks fewer coordinates.
Compare residuals on new points, solution error, and runtime.
An inverse problem learns a physical parameter
inverse: measurements of u
→ estimate α and learn u
What must change?
Represent positive α with a trainable parameter.
Include it in the optimizer and residual calculation.
Use observations that contain enough information to identify α.
Diagnose a run that does not learn
| symptom | check |
|---|---|
| outputs all near zero | initial loss, its weight, and targets |
| physics loss stays large | autograd graph, tanh, sampling, learning rate |
| shape mismatch or strange loss | all x,t,u arrays are (N,1) |
| fits data but violates physics | PDE weight and residual on new points |
| data conflicts with the PDE | α and units match the measurements |
Check your understanding: which points need labels?
Question
You sample (x,t)=(0.37,0.64) to enforce the heat equation.
Do you need to know its true temperature before calculating the physics loss?
No. Predict û, differentiate it, then compare û_t−αû_xx with zero.
This is a collocation point; its target is a residual of zero.
Check your understanding: why does the zero model fail?
Question
The network predicts û=0 everywhere.
It obeys the PDE and both boundaries. Which loss rejects it?
The initial loss: at x=0.5,t=0, the target is 1.
Interior data also rejects zero wherever the observed temperature is positive.
Keep these terms distinct
| term | meaning in this example |
|---|---|
| θ / parameters | network weights and biases |
| α / diffusivity | known physical coefficient |
| û / prediction | network's temperature estimate |
| r / residual | violation of the heat equation |
| loss / objective | weighted scalar to minimize |
| gradient | how output or loss changes with an input |
When are PINNs a useful choice?
Useful applications
Sparse measurements with known physics.
Estimating unknown coefficients.
A smooth function that can be queried and differentiated.
Practical limitations
Training can be difficult for stiff or multiscale equations.
Low sampled loss gives no universal accuracy guarantee.
Traditional numerical solvers are often faster and more reliable.
Continue with primary references
Your next step: train, predict, and explain
- Download the three files and run the short check.
- Train for 2,500 steps and inspect both saved figures.
- Predict u(0.5,0.5) and compare with 0.610498.
- Run the physics-only experiment and explain the result.
Validate your implementation before training
python pinn_heat_equation.py --self-test # Expected output: # 4 checks passed
What is checked
The exact solution's PDE residual, prediction shape, parameter count, and finite gradients through the full loss.
Training also rejects nonfinite gradients and parameters.