Index Ai Academy
Glossary · Phase 11 · Professional practice

Annotation guideline

Also written: annotation guidelines, labeling guideline

The written rules annotators follow when they label, with examples for the hard cases.

Try it now

Rows are Ana's label, columns are Ben's. The lit cells are the confused pair.

Learn it properly

Labels and annotator agreement · Labels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score. About 45 minutes.

Other terms from the same lesson

  • adjudication: Having an expert decide the label of an item on which the annotators disagreed.
  • annotator: A person who reads items and gives each one a label, following a written guideline.
  • gold question: An item with a known right answer, mixed into the labeling queue unmarked, used to estimate an annotator's accuracy.
  • label noise: The share of labels in a data set that are wrong. Scattered mistakes do less harm to a model than mistakes that all lean one way.
  • raw agreement: The share of items on which two annotators gave the same label. It is high by chance alone when one label is common.
Keep learning

More in plain English.

1286 terms in the glossary, each with the lesson that teaches it.

1286 terms

The glossary

Every word you'll meet in AI and machine learning, explained in everyday language.

Phase 11 · lesson 45 min

Labels and annotator agreement

Labels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write