Math Toolbox · One focused lesson
An old average plus a fraction of its error can equal a freshly recomputed average. This pattern connects several learning methods.
Reading tips and color key
Start with the explanation and experiment. Open the symbol cards when you want the details. Teal: probabilities; blue: states; orange: actions; green: rewards; purple: values; gray: parameters and settings; crimson: losses and errors. Written labels give every symbol its role.
A noisy measurement can still guide a useful step
Three measurements have average 4, so their total is 12. A new measurement of 8 makes the total 20 across four measurements: the new average is 5. Equivalently, move one quarter of the way from the old average toward the new measurement.
Rebuild the total
Old total: 3 × 4 = 12
Add the measurement: 12 + 8 = 20
20 ÷ 4 = 5
Correct the old average
Gap: 8 − 4 = 4
Move one quarter: ¼ × 4 = 1
4 + 1 = 5
Equation anatomy · update an arithmetic mean without storing every sample · show / hide
| Symbols and jobs | Type and randomness | Example |
|---|---|---|
| \({\color{#7542b0}{m}}_{{\color{#525968}{n}}-1},{\color{#7542b0}{m}}_{\color{#525968}{n}}\): old and new sample means | Real estimates; vary across sampled datasets | Old mean 4; new mean 5 |
| \({\color{#7542b0}{x}}_{\color{#525968}{n}}\): newest measurement | Real observed value; fixed once recorded | Fourth measurement is 8 |
| \({\color{#525968}{n}},1/{\color{#525968}{n}}\): new sample count and step size | Integer n ≥ 2 in this derivation; chosen scalar 0–1 | n = 4, step = ¼; initialize m₁ = x₁ |
Stochastic approximation uses noisy observations to make successive corrections. Replacing 1/n with a positive step size α changes how evidence is weighted. The 1/n schedule gives every recorded measurement equal weight. A constant α between 0 and 1 gradually forgets older measurements, helping track changes; persistent noise generally keeps it fluctuating around a fixed mean.
Why shrinking steps need two conditions
For independent, identically distributed samples with finite variance, a finite starting estimate, and deterministic positive steps, these standard conditions give convergence of the mean update with probability one:
Equation anatomy · a common stochastic-approximation step schedule · show / hide
| Symbol | Meaning |
|---|---|
| \({\color{#525968}{\alpha}}_{\color{#525968}{n}}\) | Deterministic positive step at update n; replaces 1/n in the correction above |
| \({\color{#525968}{n}},\sum,\infty\) | Positive integer update index; add all terms; consider indefinitely many updates |
The first condition leaves enough total movement to correct an initial error. The second limits accumulated squared step sizes, helping average out noise. The schedule 1/n satisfies both; 1/n² shrinks too quickly for the first, while a constant step fails the second. General root-finding also needs stability and suitable noise assumptions; step schedules alone are insufficient.
This explains why sample averages, SGD, and TD can share a correction pattern. Their targets and guarantees still differ. A finite run budget is a stopping choice, separate from a theorem about infinitely many updates.
Connect the dial to reinforcement learning
A policy network’s parameters control action probabilities. Policy-gradient methods estimate how expected return changes with those parameters, using sampled experience. A critic can supply learned value estimates. To maximize return, follow an estimated ascent direction; equivalently, minimize its negative.
There are two layers: choosing a learning objective or estimator, then optimizing it. PPO defines a policy surrogate and commonly updates it with minibatches. SGD is an optimizer; PPO is an RL method that uses optimization. A better surrogate value does not guarantee a better return. Original PPO paper.
General Math/ML toolbox: functions → derivatives → chain rule → gradients → optimization → stochastic estimates. RL application: value functions → policy gradient and actor–critic. Start smaller with the bandit lab or inspect sequential choices in the grid lab.
Be First to Comment