The glossary
Every word you'll meet in AI and machine learning, explained in everyday language.
An item with a known right answer, mixed into the labeling queue unmarked, used to estimate an annotator's accuracy.
A vote treats every annotator alike. To find out whether one of them is far worse than the rest, use gold questions: items whose right answer is already known, settled by an expert, and mixed into the queue unmarked. An annotator's score on the gold questions estimates their accuracy on everything else.
Rows are Ana's label, columns are Ben's. The lit cells are the confused pair.
Labels and annotator agreement · Labels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score. About 45 minutes.
1286 terms in the glossary, each with the lesson that teaches it.
Every word you'll meet in AI and machine learning, explained in everyday language.
Labels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write