Solving systems by elimination
Why this matters
Lesson 22 solved two equations in two unknowns by eliminating one letter. Real problems have more: three unknowns, ten, a thousand. Elimination scales up by writing only the numbers, in a matrix, and following one method: clear the numbers below the diagonal, row by row, until the matrix is a staircase, then read the answer off from the bottom up. It's the method behind np.linalg.solve, it shows plainly when a system has no answer or endless ones, and it answers a question at the heart of ML: which weights make a model fit these examples exactly, and what to do when no weights can.
- Write a system of equations as an augmented matrix [A | b]
- Use row moves to reach a staircase, then substitute back up to the answer
- Tell the three outcomes apart: one solution, none, or infinitely many
- Solve for a model's weights exactly, in NumPy and PyTorch, and see why more examples than weights needs a best fit instead
The math: row moves down to a staircase
Purpose. Many equations, many unknowns, one method.
Write each equation as a row of its coefficients, with the right-hand side after a bar. The system
x + y + z = 6, 2x + 3y + z = 11, x + 3y + 4z = 19
becomes the augmented matrix:
In plain words: each row is one equation; each column before the bar is one unknown.
Three row moves keep the solution the same: swap two rows; multiply a row by a nonzero number; add a multiple of one row to another. Each is something you could do to the equations themselves without changing what solves them.