Reinforcement Learning · One focused lesson
When transition chances and rewards are known, use the rulebook to improve a plan.
Reading tips and color key
Start with the explanation and experiment. Open the symbol cards when you want the details. Teal: probabilities; blue: states; orange: actions; green: rewards; purple: values; gray: parameters and settings; crimson: losses and errors. Written labels give every symbol its role.
Dynamic programming: learn from a model
Why use it? A trustworthy small-world rulebook lets us compare actions before trying them.
How does it work? Back up reward plus discounted next value. Value iteration uses a maximum; policy iteration alternates evaluation and improvement.
What does it give you? Values and a strategy derived from the model. The equation below explains the best-action backup.
When does it struggle? Every sweep may inspect many states and transitions. A wrong model gives wrong plans; sampled methods are useful when the rulebook is unavailable.
Suppose someone hands you the maze’s rulebook: all movement chances and expected rewards. You can reason about possible moves without physically trying each one.
Policy evaluation repeatedly applies the Bellman expectation calculation using a fixed policy and the latest value estimates. Policy improvement changes the policy toward better-valued actions. Policy iteration alternates these two jobs. Value iteration repeatedly uses the best action’s backup, moving toward optimal values.
Equation anatomy · Bellman optimality · show / hide
| Symbol | Meaning and kind | Form | Random? | Possible values | In code |
|---|---|---|---|---|---|
| \({\color{#172337}{\color{#7542b0}{V}}^*,{\color{#7542b0}{V}}^*({\color{#1d4ed8}{s}}),{\color{#7542b0}{V}}^*({\color{#1d4ed8}{s}}’)}\) | Optimal value function and two entries; star means optimal | Function; scalar outputs | Fixed under the model/objective | Real finite expected returns | newV[s] = max(actionBackups) approximates it |
| \({\color{#172337}{\color{#1d4ed8}{s}},{\color{#1d4ed8}{s}}’,{\color{#1d4ed8}{\mathcal{S}}}}\); \({\color{#172337}{\color{#9a4700}{a}},{\color{#9a4700}{\mathcal{A}}}}\) | Current/next states and set; action and set | Labels; finite sets | Fixed candidates | Valid states/actions | Loop through states and actions |
| \({\color{#172337}{\color{#006e78}{P}}({\color{#1d4ed8}{s}}’\mid {\color{#1d4ed8}{s}},{\color{#9a4700}{a}})}\); \({\color{#172337}{\color{#087444}{r}}({\color{#1d4ed8}{s}},{\color{#9a4700}{a}},{\color{#1d4ed8}{s}}’)}\) | Transition chance; conditional reward mean | Scalar function outputs | Fixed model quantities | Probability 0–1; reward real | p*(r + gamma*V[next]) |
| \({\color{#172337}{\color{#525968}{\gamma}}}\); \({\color{#172337}{\max,\sum,\in,\mid}}\) | Discount; largest value, sum, membership, given | Scalar setting; operators/notation | Fixed | Discount 0–1; maximum returns a number | Math.max(...actionBackups) |
The fixed-policy equation averages over the policy’s choices. The optimality equation takes the best available action. Max returns its value; argmax returns an action achieving that value. Model access is a real advantage: compare planning sweeps and sampled experience with their information budgets stated.
Be First to Comment