Index Ai Academy
Glossary · Phase 3 · Machine learning

Value iteration

Finding the best policy by sweeping the Bellman equation over every state, taking the best action each time, until the values stop changing.

Try it now

A reward k moves away is worth gamma to the power k of its face value today.

Learn it properly

Decisions in sequence: states, rewards and value · A Markov decision process on a grid world: discounted return, the value of a state, the Bellman equation, and value iteration to the best policy. About 40 minutes.

Other terms from the same lesson

  • Bellman equation: The value of a state is the reward for the next move plus the discount factor times the value of the state that move leads to.
  • discount factor: A number between 0 and 1, written gamma, that a reward is multiplied by for every step it arrives later. It sets how far ahead an agent cares.
  • episode: One run of an agent from a starting state to an end state.
  • Markov decision process: A problem described by states, actions, rewards and where each action leads, in which the next state depends only on the current state and action.
  • policy: A rule that picks an action for every state an agent can be in. When a language model is trained with rewards, the model itself is the policy.
  • return: In reinforcement learning, the total reward collected from a point on, often with later rewards discounted.
Keep learning

More in plain English.

1286 terms in the glossary, each with the lesson that teaches it.

1286 terms

The glossary

Every word you'll meet in AI and machine learning, explained in everyday language.

Phase 3 · lesson 40 min

Decisions in sequence: states, rewards and value

A Markov decision process on a grid world: discounted return, the value of a state, the Bellman equation, and value iteration to the best policy.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write