Reinforcement Learning · One focused lesson
What is a situation worth under a strategy? Build the answer from the next reward and the value of what follows.
Reading tips and color key
Start with the explanation and experiment. Open the symbol cards when you want the details. Teal: probabilities; blue: states; orange: actions; green: rewards; purple: values; gray: parameters and settings; crimson: losses and errors. Written labels give every symbol its role.
06 · Put a number on a situation
Value: what happens if we keep following this strategy?
A tile beside the flag looks promising. But its usefulness also depends on your strategy. An explorer who keeps walking away from the flag gets a different future from one who heads toward it.
State value V asks about starting in a state. Action value Q asks about taking a particular first action there. Both then follow the specified policy. The value is an expected return, so it averages over possible futures.
Equation anatomy · two questions about the same future · show / hide
| Symbol | Meaning and kind | Form | Random? | Possible values | In code |
|---|---|---|---|---|---|
| \({\color{#172337}{\color{#7542b0}{V}}^{\color{#006e78}{\pi}},{\color{#7542b0}{Q}}^{\color{#006e78}{\pi}}}\) | State-value and action-value functions | State → scalar; (state, action) → scalar | Deterministic for fixed model and policy | Real expected returns, if finite | V[state], Q[state][action] |
| \({\color{#172337}{\color{#006e78}{\pi}}}\) | Policy: rule for action probabilities | State → action distribution | Fixed distribution; action samples can vary | Action probabilities 0–1, summing to 1 | policy[state] = [0.1,0.7,0.1,0.1] |
| \({\color{#172337}{\color{#1d4ed8}{S}}_{\color{#525968}{t}},{\color{#1d4ed8}{s}}}\); \({\color{#172337}{\color{#9a4700}{A}}_{\color{#525968}{t}},{\color{#9a4700}{a}}}\) | State/action variables and their specified values | IDs in the grid | Uppercase variables random; conditioned values fixed | Valid states and actions | state = 12; action = 1 |
| \({\color{#172337}{\color{#087444}{G}}_{\color{#525968}{t}}}\); \({\color{#172337}{\color{#525968}{t}}}\); \({\color{#172337}{\mathbb{E}_{\color{#006e78}{\pi}},\mid}}\) | Return; time; average under policy and environment, given conditions | Scalar; index; operator/notation | Return generally random; expectation deterministic | Real return; nonnegative time | Average many futures from the specified start |
The whole object is not one entry. On a grid with 36 possible state IDs and four actions, V can be a 36-number vector and Q a 36 × 4 matrix. Q[12][1] is just one number. A policy row [0.1,0.7,0.1,0.1] is a whole action distribution; its second entry, 0.7, is one probability. The dot in \({\color{#172337}{\color{#006e78}{\pi}}(\cdot\mid {\color{#1d4ed8}{s}})}\) means “include every action.”
An estimate stored after training is not automatically the true value. Different data produce different estimates. We use a hat for estimates when that distinction matters, and plain V or Q in updates to keep the writing readable.
07 · A big future, split into a small step
The Bellman equation
You do not have to list every possible journey at once. Ask what you earn on the next move, then add how promising the next tile is. Average that answer over the moves your strategy chooses and the outcomes the world allows.
The Bellman expectation equation applies the return’s one-step decomposition to a fixed policy’s value. It is a consistency relationship between true values; an algorithm can use it to improve estimates.
2
0.9 × 10 = 9
Discount later points11
The bar makes one certain future. With several possible futures, build this sum for each branch, multiply by its chance, and add. The expectation symbol below is shorthand for that weighted averaging.
Equation anatomy · reward now plus discounted next value, averaged · show / hide
| Symbol | Meaning and kind | Form | Random? | Possible values | In code |
|---|---|---|---|---|---|
| \({\color{#172337}{\color{#7542b0}{V}}^{\color{#006e78}{\pi}}({\color{#1d4ed8}{s}}),{\color{#7542b0}{V}}^{\color{#006e78}{\pi}}({\color{#1d4ed8}{S}}_{{\color{#525968}{t}}+1})}\) | Current value and value at the next state | Scalar function evaluations | Current value fixed; next lookup varies with random input | Real expected returns | V[state], V[next] |
| \({\color{#172337}{\color{#087444}{R}}_{{\color{#525968}{t}}+1}}\); \({\color{#172337}{\color{#1d4ed8}{S}}_{\color{#525968}{t}},{\color{#1d4ed8}{S}}_{{\color{#525968}{t}}+1},{\color{#1d4ed8}{s}}}\) | Next reward; current and next states; specified start | Scalar reward; state IDs here | Reward and next state may be random; s fixed | Real reward; valid state IDs | transition = step(state,action) |
| \({\color{#172337}{\color{#006e78}{\pi}}}\); \({\color{#172337}{\mathbb{E}_{\color{#006e78}{\pi}},\mid}}\) | Policy; average over action/outcome randomness, given the start | Function; expectation operator; notation | Policy and expectation fixed for the model | Policy outputs a normalized distribution | Probability-weight all allowed outcomes |
| \({\color{#172337}{\color{#525968}{\gamma}},{\color{#525968}{t}}}\) | Discount and time index | Scalar setting; integer | Fixed | \({\color{#172337}{0\le{\color{#525968}{\gamma}}\lt 1;\ {\color{#525968}{t}}\ge0}}\) here | gamma = 0.9 |
With one certain outcome, reward 2, discount 0.9, and next value 10, the current value is 11. With uncertain outcomes, average their reward-plus-next-value answers. A single observed transition is a sample of the Bellman right-hand side, not usually its exact average.
Open the expectation into probability sums
For a finite MDP, first consider every action. Then consider every next state for that action. Weight each possibility by the policy’s action probability and the environment’s transition probability.
Equation anatomy · two sources of chance · show / hide
| Symbol | Meaning and kind | Form | Random? | Possible values | In code |
|---|---|---|---|---|---|
| \({\color{#172337}{\color{#1d4ed8}{s}},{\color{#1d4ed8}{s}}’,{\color{#1d4ed8}{\mathcal{S}}}}\); \({\color{#172337}{\color{#9a4700}{a}},{\color{#9a4700}{\mathcal{A}}}}\) | Current/next state, state set; action, action set | Fixed labels and finite sets | Summation candidates are fixed | Valid states/actions | for (next of states) |
| \({\color{#172337}{\color{#006e78}{\pi}}({\color{#9a4700}{a}}\mid {\color{#1d4ed8}{s}}),{\color{#006e78}{P}}({\color{#1d4ed8}{s}}’\mid {\color{#1d4ed8}{s}},{\color{#9a4700}{a}})}\) | Action-choice chance; next-state chance | Scalar entries of separate distributions | Deterministic for fixed policy/model | 0–1; respective rows sum to 1 | policy[s][a]*P[s][a][next] |
| \({\color{#172337}{\color{#087444}{r}}({\color{#1d4ed8}{s}},{\color{#9a4700}{a}},{\color{#1d4ed8}{s}}’)}\) | Expected immediate reward conditioned on this transition | Function returning one scalar | Fixed conditional mean, even if rewards fluctuate | Real numbers | expectedReward[s][a][next] |
| \({\color{#172337}{\color{#7542b0}{V}}^{\color{#006e78}{\pi}}({\color{#1d4ed8}{s}}),{\color{#7542b0}{V}}^{\color{#006e78}{\pi}}({\color{#1d4ed8}{s}}’)}\); \({\color{#172337}{\color{#525968}{\gamma}}}\) | Current/next true values; discount | Scalar function outputs; setting | Fixed | Real finite values; discount below 1 here | rewardMean + gamma*V[next] |
| \({\color{#172337}{\sum,\in,\mid}}\) | Add candidates, set membership, conditioning | Operators/notation | No | Two finite weighted sums | Add every probability-weighted term |
Using the conditional mean reward makes this form valid even when reward and next state are related. For an impossible transition, its zero probability removes its contribution. Terminal states have value zero; their arriving reward is counted on entry.
Be First to Comment