Multi-layer networks

Many layers: the multi-layer perceptron

A multi-layer perceptron chains layers together. "Perceptron" is an early name for an artificial neuron. The inputs go into a first layer. Its outputs become the inputs of the next layer, and so on. The last layer gives the answer.

The layers in the middle are called hidden layers. "Hidden" only means their values are neither the inputs nor the final answer. There's nothing secret about them.

The activation between layers matters. Without it, two layers collapse into a single layer. You saw that in the activation functions lesson.

XOR with one hidden layer

Many hallways have a light with a switch at each end. With both switches down, the light is off. Flip either one and the light comes on. Flip the other too and it goes off again. The light is on when exactly one switch is up. That rule is called XOR ("exclusive or").

In the neuron lesson, one neuron failed at XOR. No straight line can split its four points correctly. Two hidden ReLU neurons are enough:

• The first neuron computes ReLU of (input 1 + input 2). It's on when at least one input is on.
• The second computes ReLU of (input 1 + input 2 − 1). It's on only when both are on.
• The output takes the first neuron and subtracts twice the second.

Try the input (1, 1) by hand. The first neuron gives 1 + 1 = 2. The second gives 1 + 1 − 1 = 1. The output is 2 − 2 × 1 = 0. Both switches are up, so the light is off. That's correct.

import numpy as np

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
W1 = np.array([[1.0, 1.0],
               [1.0, 1.0]])           # 2 inputs → 2 hidden neurons
b1 = np.array([0.0, -1.0])
W2 = np.array([[1.0], [-2.0]])       # 2 hidden → 1 output
b2 = np.array([0.0])

H = np.maximum(0, X @ W1 + b1)       # hidden layer
out = H @ W2 + b2                    # output layer
for x, h, o in zip(X, H, out):
    print(f"input {x}  hidden {h}  output {o[0]:.0f}")

The outputs are 0, 1, 1 and 0. That's XOR.

Now look at the hidden column. The hidden layer gives each point a new description. The inputs (0, 1) and (1, 0) both become (1, 0). The input (1, 1) becomes (2, 1). In this new description, one straight line can split the two answers. The output neuron draws that line.

This is what hidden layers do in general. They redescribe the data until the answer becomes simple.

Learning the bends from data

You picked the XOR weights by hand. Real networks learn their weights from data, with the gradient descent you met in Phase 3.

The code below makes 300 points in two curved shapes that hook into each other, like two half moons. The noise setting scatters the points a little. Two models then learn to tell the moons apart:

• Logistic regression, which is one neuron. It can only cut with a straight line.
• An MLP with 2 inputs, two hidden layers of 16 neurons each, and 1 output.

The scikit-learn library does the training. max_iter=3000 lets the MLP take up to 3,000 passes over the data. random_state=0 fixes the random choices, such as where the points land and the network's starting weights. So you get the same result every run.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_moons
from sklearn.linear_model import LogisticRegression
from sklearn.neural_network import MLPClassifier

X, y = make_moons(n_samples=300, noise=0.2, random_state=0)
models = {"one neuron (logistic regression)": LogisticRegression(),
          "MLP: 2 → 16 → 16 → 1": MLPClassifier(hidden_layer_sizes=(16, 16), max_iter=3000, random_state=0)}

gx, gy = np.meshgrid(np.linspace(-1.6, 2.6, 150), np.linspace(-1.1, 1.6, 150))
fig, axes = plt.subplots(1, 2, figsize=(10, 3.6))
for ax, (name, m) in zip(axes, models.items()):
    m.fit(X, y)
    zz = m.predict_proba(np.c_[gx.ravel(), gy.ravel()])[:, 1].reshape(gx.shape)
    ax.contourf(gx, gy, zz, levels=[0, 0.5, 1], alpha=0.25, cmap="coolwarm")
    ax.scatter(X[:, 0], X[:, 1], c=y, cmap="coolwarm", s=10)
    ax.set_title(name)
    print(f"{name:<34} training accuracy {m.score(X, y):.2f}")

The shaded areas show which class each model would pick at every spot. The single neuron's border is a straight line, so it gets points near the tips of the moons wrong. The MLP's border bends to follow the moons, and its training accuracy is higher. Training accuracy is measured on the same points the model learned from. Later lessons test on new data.

Counting parameters

A network's parameters are all the numbers it learns: every weight and every bias. You can count them layer by layer. Each output neuron has one weight per input, plus its own bias.

In plain words: each neuron in the layer needs one number per input, plus one for its bias. Multiply that by how many neurons the layer has.

Try a small network: 2 inputs, 3 hidden neurons, 1 output. The first layer has (2 + 1) × 3 = 9 parameters. The second has (3 + 1) × 1 = 4. The total is 13.

The code does the same count for a classic digit reader. Its images are 28 × 28 pixels, which makes 784 inputs. Then come 128 hidden neurons and 10 outputs, one per digit. Change sizes to [2, 3, 1] and it prints 13.

sizes = [784, 128, 10]          # a classic digit classifier: 28×28 pixels → 128 hidden → 10 digits
total = 0
for n_in, n_out in zip(sizes, sizes[1:]):
    layer = (n_in + 1) * n_out
    total += layer
    print(f"{n_in:>4} → {n_out:<4} weights {n_in * n_out:>7,} + biases {n_out:>4} = {layer:>7,}")
print(f"total: {total:,} parameters")

The first layer has 100,480 parameters and the second has 1,290. That's 101,770 in all. zip(sizes, sizes[1:]) pairs each layer's input count with its output count.

inputs, outputs = 3, 2
print(inputs * outputs + outputs)

How big real networks get

For scale, GPT-2 small, which is a 2019 language model from OpenAI, has about 124 million parameters. The biggest LLMs today have hundreds of billions.

Each transformer block in an LLM contains an MLP. It typically widens the data to 4 times the model's width, then narrows it back. In that setup, those two weight matrices hold about two-thirds of the block's parameters.

Width, depth and what MLPs can do

A network's width is how many neurons a layer has. Its depth is how many layers it has.

The universal approximation theorem is a famous result about width. It says a network with one hidden layer can get as close as you like to any continuous function, given enough neurons. A continuous function is a rule whose output has no sudden jumps.

Here's a simplified picture of why. Each ReLU neuron adds one bend to the output. With enough bends, you can trace any smooth curve, like drawing a circle with many short straight strokes.

"Enough" can be far too many to be practical, though. Depth helps. Each layer builds on the features of the one before, the way letters make words and words make sentences. A deep network can often represent a complicated function with far fewer neurons than a wide, shallow one.

The theorem promises that a good network exists. It doesn't promise that training will find it. The rest of this phase is about finding it.