One neuron, start to finish
Why this matters
This module adds no new math. Every formula in it was taught and practiced in Modules 1 to 7; what's new is running them together, the way a real system does. It starts with the smallest thing that learns: one neuron. Its training step is five familiar pieces, a weighted sum, an activation, a loss, slopes by the chain rule and an update, and once they run as one loop, the neuron learns. You'll take that step by hand, in NumPy and in PyTorch, then see exactly what one neuron can and cannot learn, which explains why networks stack them: the next lesson.
- Follow one neuron through a full training step by hand: weighted sum, activation, loss, slopes, update
- Name the earlier lesson behind each of the five steps
- Write the five steps as five lines of NumPy, then loop them until the loss reaches a target
- Take the same step in PyTorch with backward() and optim.SGD, and get the hand numbers back
- See what a single neuron can learn, a split of the plane by a straight line, and what it can't
The math: five steps, all known
Purpose. Run one complete training step of one neuron by hand.
A neuron with two inputs has two weights w₁ and w₂ and a bias b. Here the inputs are x = (2, 1), the weights (0.5, −1), the bias 0.5, and the target, the answer it should give, is y = 1. One training step:
- Weighted sum (lesson 50): z = w₁x₁ + w₂x₂ + b = 1 − 1 + 0.5 = 0.5.
- Activation (lesson 71): a = relu(z) = max(0, z) = 0.5.
- Loss (lesson 79): L = (a − y)² = (0.5 − 1)² = 0.25.
- Slopes (lessons 74 and 81): backward through each step, multiplying local slopes.
- Update (lesson 79): every weight moves against its slope, by the learning rate.
Steps 1 to 3 are the forward pass, step 4 the backward pass. Step 4 is the only one with any work in it, and it's lesson 81's graph again: seven small steps, walked backward.