The forward pass

import math

x1, x2 = 1.0, 2.0
h1 = max(0.0, 0.5 * x1 + 0.25 * x2 + 0.0)
h2 = max(0.0, -1.0 * x1 + 0.5 * x2 + 0.5)
o1 = 1.0 * h1 - 1.0 * h2
o2 = -0.5 * h1 + 3.0 * h2
print("hidden:", h1, h2)
print("logits:", o1, o2)

e1, e2 = math.exp(o1), math.exp(o2)
p1, p2 = e1 / (e1 + e2), e2 / (e1 + e2)
print("probabilities:", round(p1, 3), round(p2, 3))
print("loss:", round(-math.log(p2), 3))

A whole batch at once

Real networks rarely see one example at a time. They take a batch: many examples pushed through together. Think of a teacher marking a whole stack of tests with one answer key open on the desk.

You stack the examples as the rows of a table called $X$. Each row is one example. Each column is one input. The layer's weights go in a table called $W_1$, with one column per neuron. The biases go in a list called $b_1$, with one bias per neuron. Now the whole layer is one matrix multiplication, which you met in Phase 2. The $H$ on the left is the table of hidden values.

In plain words: for every example and every neuron, do the weighted sum plus bias. Then replace any negative result with 0. It's the one-neuron recipe, done for all rows and all neurons at once.

Here is the stepper's hidden layer written this way. Row 1 is the stepper's input, (1, 2). Row 2 is a new input, (3, 2). For row 2, neuron 1 gives 0.5 × 3 + 0.25 × 2 + 0 = 2.0. Neuron 2 gives −1 × 3 + 0.5 × 2 + 0.5 = −1.5, and ReLU turns that into 0.

import numpy as np

X = np.array([[1.0, 2.0],      # example 1: the inputs from the stepper
              [3.0, 2.0]])     # example 2: a new pair of inputs
W1 = np.array([[0.5, -1.0],    # column 1 holds neuron 1's weights,
               [0.25, 0.5]])   # column 2 holds neuron 2's weights
b1 = np.array([0.0, 0.5])      # one bias per neuron

H = np.maximum(0, X @ W1 + b1) # weighted sums for both examples, then ReLU
print(H)
print(H.shape)

The first row matches the stepper: 1.0 and 0.5. NumPy prints 1.0 as 1. to save space. The second row is 2.0 and 0, as worked out above.

A real dataset

Now a bigger case. The Iris dataset has 150 flowers. Each has 4 measurements, such as petal length, and belongs to one of 3 species. The code takes 8 of the flowers. It runs them through a 4 → 16 → 3 network: 4 inputs, 16 hidden neurons and 3 output classes. The weights are small and random, so the network is untrained.

Two details in the code are new. First, it standardizes the inputs. Each measurement is rescaled so that, across all 150 flowers, its average is 0 and its standard deviation is 1. That stops one large measurement from drowning out the others. Second, P[np.arange(len(y)), y] picks out, for each flower, the probability given to its true species.

import numpy as np
from sklearn.datasets import load_iris

data = load_iris()
X, y = data.data[::19], data.target[::19]                     # 8 flowers spread over the 3 species
X = (X - data.data.mean(axis=0)) / data.data.std(axis=0)      # standardize the inputs

rng = np.random.default_rng(0)
W1, b1 = rng.normal(0, 0.1, size=(4, 16)), np.zeros(16)       # small random starting weights
W2, b2 = rng.normal(0, 0.1, size=(16, 3)), np.zeros(3)

Z1 = X @ W1 + b1                     # (8, 4) @ (4, 16) → (8, 16)
H = np.maximum(0, Z1)                # ReLU, same shape
logits = H @ W2 + b2                 # (8, 16) @ (16, 3) → (8, 3)
P = np.exp(logits - logits.max(axis=1, keepdims=True))
P = P / P.sum(axis=1, keepdims=True) # softmax per row: (8, 3), each row sums to 1

for name, arr in [("X", X), ("Z1 / H", H), ("logits", logits), ("P", P)]:
    print(f"{name:<7} shape {arr.shape}")
print("\nprobabilities for the first 3 flowers:\n", P[:3].round(3))
loss = -np.mean(np.log(P[np.arange(len(y)), y]))
print(f"\ncross-entropy loss: {loss:.3f}   (random guessing among 3 classes gives ln 3 = {np.log(3):.3f})")

With small random weights, the untrained network gives every species about a third. It has learned nothing yet, so it spreads its bets evenly. Its loss lands close to ln 3, which is the loss of guessing evenly among 3 classes, because $-\ln \tfrac{1}{3} = \ln 3$. Training will push the loss down by changing the weights. The next lessons show how to work out which way to change each one.

Tip: an untrained classifier's first loss should be close to ln of the number of classes. If it starts much higher, look for a bug or starting weights that are too big.

Shapes are the first thing to check

Every array has a shape: its size along each direction. A shape of (8, 16) means 8 rows and 16 columns. Most bugs in neural-network code show up as a shape you didn't expect. So print shapes often.

The key rule for matrix multiplication is that the inner sizes must match. (8, 4) @ (4, 16) works because both inner numbers are 4. The result keeps the outer sizes, so it's (8, 16).

The bias $b_1$ has shape (16,), one number per neuron. NumPy broadcasts it. That means it adds the same list of biases to every row, so every example gets the same biases.

Stable softmax

Computers store numbers with a limited size. $e^{1000}$ is far too big, so it comes out as infinity, and softmax breaks. Real models can produce large logits, so this matters.

The fix is to subtract the row's largest score from every score before you take powers of $e$. The probabilities come out exactly the same. Think of measuring everyone's height against the tallest person in the room. The numbers change, but the gaps between people don't. Softmax only cares about the gaps.

After the shift, the biggest score is 0 and the rest are negative. So every power of $e$ is at most 1, and nothing overflows. Every real library does this. In the batch code above, it's the logits - logits.max(axis=1, keepdims=True) part.

import math

def softmax(scores):
    top = max(scores)
    exps = [math.exp(s - top) for s in scores]
    total = sum(exps)
    return [round(e / total, 3) for e in exps]

print(softmax([2.0, 1.0]))
print(softmax([1002.0, 1001.0]))

Both lines print [0.731, 0.269]. The gap is 1 in both cases, so the answer is the same. Without the shift, the second line would try to compute $e^{1002}$ and fail.

Logits in an LLM

A language model's forward pass ends the same way. For each position in the text, it produces a list of logits, one for each token in its vocabulary. That's often 100,000 tokens or more. Softmax turns them into probabilities for the next token. The "temperature" setting you met in Phase 1 divides those logits before the softmax.