Adding and scaling vectors
Why this matters
Vectors can be added, subtracted and stretched, and each operation has a picture: walking one arrow then another, or making an arrow longer, shorter or backward. These few moves are the whole of how a model learns. Every training step takes the weights, a vector, and subtracts a small multiple of another vector, the gradient. This lesson builds those moves by hand, in NumPy, and then as that one line of learning. Along the way come the arrow from one point to another, b − a, and the mix of two vectors, which is how a point partway between two others is found and how word vectors can be blended and compared.
- Add and subtract vectors component by component, and picture it tail to head
- Scale a vector by a number, to stretch, shrink or reverse it, and mix two vectors
- Do vector arithmetic on NumPy arrays, know that sizes must match, and see why one array line beats a loop (vectorization)
- Read a training step, w ← w − η·g, as one vector subtraction, and run it in PyTorch, by hand and with an optimizer
The math: combining and resizing arrows
Purpose. Combine movements, and make them bigger, smaller or backward.
Adding. Add matching components: (3, 1) + (1, 2) = (4, 3). As arrows, put the second arrow's tail at the first one's head (its tip), tail to head; the sum goes from the first tail to the last head. Order doesn't matter: either way round you land in the same place.
Scaling. Multiply every component by one number, a scalar: 2 × (3, 1) = (6, 2), twice as long, same direction. A scalar between 0 and 1 shrinks it; a negative scalar reverses it: −1 × (3, 1) = (−3, −1).
Subtracting. b − a is the arrow that goes from a to b: what you'd add to a to reach b.