The glossary
Every word you'll meet in AI and machine learning, explained in everyday language.
A transformer whose weights were produced from a small program by a compiler, not by training. Its computation is known exactly, which makes it a test bench for interpretability tools.
What you did by hand has been automated for transformers. RASP is a small programming language whose operations match what attention heads and feed-forward layers can do. Tracr, released in 2023, is a compiler that takes a RASP program and produces the actual weights of a transformer that runs it. Researchers use those models as a laboratory: when you know exactly what the weights compute, you can test whether an interpretability method finds it.
The "follow" layer as a picture. Two whole columns are blank: this policy never looks at how high the ball is or which way it is moving.
Writing weights by hand · Set a small network's 55 weights so it catches every ball in a paddle game, the way a program would be written, and meet compiled transformers. About 40 minutes.
1286 terms in the glossary, each with the lesson that teaches it.
Every word you'll meet in AI and machine learning, explained in everyday language.
Set a small network's 55 weights so it catches every ball in a paddle game, the way a program would be written, and meet compiled transformers.
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write