Index Ai Academy
Glossary · Phase 6 · Modern LLM engineering

Sign test

Also written: sign tests

A check for luck when two versions are scored on the same items. It looks only at the items that changed, and asks how often a fair coin would give a split that lopsided.

Try it now

Learn it properly

Did it work? Testing an edit · Review a patch for a tiny GPT that can't stop talking: held-out questions, a paired comparison, a same-size control, and what the edit cost. About 40 minutes.

Other terms from the same lesson

  • control edit: An edit of the same size and form as the one being tested, with its aim replaced by chance. If it works as well, the real edit's explanation is unsupported.
Keep learning

More in plain English.

1286 terms in the glossary, each with the lesson that teaches it.

1286 terms

The glossary

Every word you'll meet in AI and machine learning, explained in everyday language.

Phase 6 · lesson 40 min

Did it work? Testing an edit

Review a patch for a tiny GPT that can't stop talking: held-out questions, a paired comparison, a same-size control, and what the edit cost.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write