Reinforcement Learning · One focused lesson
Compare an action with the usual result, then see why policy updates use probability ratios.
Reading tips and color key
Start with the explanation and experiment. Open the symbol cards when you want the details. Teal: probabilities; blue: states; orange: actions; green: rewards; purple: values; gray: parameters and settings; crimson: losses and errors. Written labels give every symbol its role.
Advantage: was this choice better than usual?
Why use it? A reward needs context: was this action better than the usual outcome from here?
How does it work? Subtract the state baseline from the action’s expected return; use estimated advantage to guide action-probability updates.
What does it give you? A signed comparison: positive means better than the baseline, negative means worse.
When does it struggle? A noisy or poorly learned baseline/critic can produce noisy advantages. An action can have positive advantage even if its absolute return is negative.
A score of 8 sounds good, but compared with what? Imagine a one-step game: choose right or left, receive points, then finish. The two choices have expected returns 8 and 2. A strategy choosing each half the time has value 5. Advantage subtracts that usual value from a move’s value. It gives the policy a direction: favor better-than-usual choices, discourage worse-than-usual ones.
Usual value: 0.5 × 8 + 0.5 × 2 = 5
8 − 5 = +3
Three points better than usual.
2 − 5 = −3
Three points worse than usual.
These are expected returns for one state under a fixed strategy, not two individual observations. The policy-weighted average advantage is zero: 0.5 × 3 + 0.5 × (−3) = 0.
Equation anatomy · compare an action with its policy’s baseline · show / hide
| Symbols and jobs | Type, randomness, range | Example |
|---|---|---|
| \({\color{#7542b0}{A}}^{\color{#006e78}{\pi}},{\color{#7542b0}{Q}}^{\color{#006e78}{\pi}},{\color{#7542b0}{V}}^{\color{#006e78}{\pi}}\) | Advantage, action-value and state-value functions; real scalar outputs, fixed for a model and policy | 8 − 5 = 3. Advantage A is a value comparison; it is not the action variable A used in trajectories. |
| \({\color{#006e78}{\pi}},{\color{#1d4ed8}{s}},{\color{#9a4700}{a}}\) | Strategy giving action probabilities; selected state and action labels; fixed in this calculation | At one tile, choose right or left with chance 0.5 each |
Connect the comparison to one policy-gradient step
Let one adjustable number z control the chance of choosing right through a sigmoid: larger z means a greater chance. After a right-hand choice, multiply its estimated advantage by the slope of its log-probability. Then move z a little in that direction. This is one sampled policy update; it does not guarantee that the whole policy improves.
Equation anatomy · this formula is for a sampled right action · show / hide
| Symbols and jobs | Type, randomness, range | Example |
|---|---|---|
| \({\color{#525968}{z}},{\color{#525968}{z}}_{\mathrm{new}},{\color{#525968}{\alpha}},{\color{#006e78}{p}}\) | Real parameter before/after; positive learning rate; right-action probability strictly between 0 and 1 | z = 0 gives p = 0.5; α = 0.1 |
| \({\color{#7542b0}{\widehat A}},{\color{#ae2455}{\widehat g}},e,\log,d/d{\color{#525968}{z}}\) | Real estimated advantage and sampled gradient contribution; exponential base, natural log and derivative. Estimates vary with data; hold advantage fixed during the actor update. | Â = 3 gives ĝ = 3 × 0.5 = 1.5 |
Follow the numbers: z: 0 → 0.15, so right’s chance: 50% → about 53.7%. For a sampled left action, the log-probability slope is −p instead. A baseline depending only on the state leaves the expected action gradient unchanged under the usual sampling assumptions and can reduce noise; it is not guaranteed to reduce variance for every choice of baseline. An actor changes action probabilities, while a critic learns values to help estimate this comparison.
A preview: PPO uses an action-probability ratio
Why use it? Repeated optimization of one batch can move a policy too far from the behavior that collected it.
How does it work? Compare new and old action probabilities and clip the surrogate’s incentive for certain large changes.
What does it give you? A tractable policy-update objective; the worked terms below show how the advantage sign matters.
When does it struggle? Clipping is not a hard bound on every probability or on KL divergence. It does not guarantee improved return; data quality and optimization still matter.
PPO compares the new and data-collecting policies at each sampled state/action. This is a single-step ratio, not the complete trajectory product. For its clipped objective, one sample contributes:
Why the sign of advantage matters · ε = 0.2
At ρ = 1.5: min(3, 2.4) = 2.4. Increasing an already favored action further earns no extra contribution here.
At ρ = 0.5: min(−1, −1.6) = −1.6. Reducing an already discouraged action further earns no extra contribution here.
Equation anatomy · a surrogate contribution to maximize · show / hide
| Symbols and jobs | Type, randomness, range | Example |
|---|---|---|
| \({\color{#006e78}{\pi}}_{\mathrm{new}},{\color{#006e78}{\pi}}_{\mathrm{old}},{\color{#006e78}{\rho}}\) | Policy functions and nonnegative ratio; fixed for current parameters and sample; denominator positive | 0.3 / 0.2 = 1.5 |
| \({\color{#1d4ed8}{s}},{\color{#9a4700}{a}},{\color{#7542b0}{\widehat A}}\) | Recorded state/action; real advantage estimate, random across datasets | Advantage +2 means better than baseline |
| \({\color{#525968}{\epsilon}},\ell,\min,\operatorname{clip}\) | Chosen scalar 0–1; real objective term; smaller-of-two and clamp operators | ε = 0.2, ratio 1.5, advantage 2 → min(3,2.4) = 2.4 |
The objective removes some incentive for favorable changes beyond the clipping threshold. It does not hard-constrain every policy ratio, guarantee a KL bound, or guarantee better return. Original PPO paper, §3.
Be First to Comment