Math Toolbox · One focused lesson
Entropy measures spread within a distribution. KL divergence compares two distributions in a specified direction.
Reading tips and color key
Start with the explanation and experiment. Open the symbol cards when you want the details. Teal: probabilities; blue: states; orange: actions; green: rewards; purple: values; gray: parameters and settings; crimson: losses and errors. Written labels give every symbol its role.
Entropy describes how spread out choices are
Why use it? Describe how concentrated a choice distribution is, and compare it with another distribution.
How does it work? Entropy averages a distribution’s surprise. KL averages the log probability ratio against a reference.
What does it give you? Two distinct diagnostic numbers: spread within one distribution, and directional difference between two.
When does it struggle? High entropy can mean random bad actions. Low KL can still hide important changes in rarely visited situations. Neither score measures task reward.
A spinner with two equally likely outcomes is less predictable than one that always selects the same outcome. Categorical entropy measures this uncertainty:
Build entropy by averaging surprise · an 80/20 spinner
Add the contributions → H(p) ≈ 0.500 nats (using unrounded values)
Equation anatomy · natural logarithms give nats · show / hide
| Symbols and jobs | Type, randomness, range | Example |
|---|---|---|
| \({\color{#006e78}{p}},{\color{#006e78}{p}}_{{\color{#525968}{i}}}\) | Fixed probability vector and entry; entries nonnegative, sum 1 | [0.5,0.5] |
| \(H,{\color{#525968}{K}},{\color{#525968}{i}},\sum,\log\) | Deterministic scalar from 0 to log K; K ≥ 1 categories, index 1…K; sum; natural log | Balanced pair: log 2 ≈ 0.693 nats; certain choice: 0 |
Use the convention 0 log 0 = 0. In RL an entropy bonus can encourage more varied action probabilities. High local entropy does not guarantee useful exploration, coverage of distant states, or better rewards. Entropy reference.
Math you can see · Sampling
Let random samples do the counting
First, estimate a circle’s area by scattering dots. Then see how to reuse samples drawn from a different distribution. Every dot, table row, and curve below comes from the experiment you run.
Equation key: probabilities · rewards · counts and settings · estimates. Chart colors identify the compared quantities and have a separate legend.
01 · Estimate by counting
How many dots land inside the circle?
A circle fits exactly inside a square. Drop points uniformly across the square: every equal-sized patch gets the same chance. The fraction inside estimates the circle’s share of the square.
Play adds 100 samples at a time and stops at the limit (at most 50,000). Pause keeps your progress. Reset applies the seed; changing D resets this experiment. Identical seeds reproduce identical points, regardless of batch size.
InsideOutside
Estimate π̂True π ≈ 3.14159
95% Wilson interval, scaled to π: —
This uses the inside/outside count as binomial data and multiplies its hit-rate interval by 4. The nominal 95% level describes approximate coverage over repeated experiments at a fixed sample count. It is not a guarantee for one run, and repeatedly checking then choosing when to stop does not preserve that coverage.
Inspect the latest 8 actual dots
| Sample | x | y | Inside? |
|---|---|---|---|
| Add dots to see their coordinates. | |||
Open the math and its anatomy
The square’s area is D². The circle’s radius is D/2, so its area is π(D/2)². Dividing gives P(inside) = π/4. Estimate that probability by c/n, then multiply by 4.
x, y ∼ Uniform(−D/2, D/2), independently
inside ⇔ x² + y² ≤ (D/2)²
π̂ = 4c/n
- D · one positive setting
- A fixed scalar length. Here 0.1 ≤ D ≤ 100. It sets both the square’s side and the circle’s diameter. Scaling both shapes does not change their area ratio.
- x, y · two sampled coordinates
- Two independent random scalars, together a length-2 vector. Each lies between −D/2 and D/2. Code draws two uniforms, shifts and scales them, then checks the squared distance.
- c, n · observed counts
- Nonnegative integer scalars; 0 ≤ c ≤ n. Increment n for every point and c only for an inside point. All points count even when older dots disappear from the display.
- π̂ · a random estimate
- A scalar in [0, 4] when n > 0. It varies across experiments and is fixed once those points are observed. Code computes 4 × c / n; no estimate is shown at n = 0.
More samples usually reduce Monte Carlo noise, but an individual next dot can move the estimate farther from π. Typical error scales like 1/√n, so roughly four times as many independent samples are needed to halve it.
Interval reference: NIST’s discussion of Wilson proportion intervals.
Ready. Add one dot to follow one sample, or 100 to see an estimate form.
02 · Correct for where samples came from
What if we sampled from the wrong mix?
Outcome A pays 1 point; outcome B pays 5 points. We want the average under the target distribution p, but our samples come from the proposal distribution q. Give each observation weight p/q to account for the mismatch.
| Outcome | Target p | Proposal q | Weight p/q | Reward |
|---|
Target average: 0.80 × 1 + 0.20 × 5 = 1.800 points. Proposal average: 4.200 points.
All three estimators use the same samples from q. Changing either probability resets this experiment using the current seed. Both outcomes keep positive probability, so sampling can reach everything the target needs.
Samples: 0
Weight ESS: —
Sum of weights: 0
Ordinary ISSelf-normalized ISNaive averageExact target average
Inspect the latest 8 actual samples
| Sample | Outcome | p | q | w = p/q | Reward | Contribution |
|---|---|---|---|---|---|---|
| Draw a sample to inspect its weight. | ||||||
Why the three estimates behave differently
wᵢ = p(Xᵢ) / q(Xᵢ)
Ordinary IS = Σᵢ wᵢrᵢ / n
Self-normalized IS = Σᵢ wᵢrᵢ / Σᵢ wᵢ
Naive average = Σᵢ rᵢ / n
Ordinary IS is unbiased for the target mean in this finite model: across repeated fixed-size experiments, its mean is exactly the target. A particular run can still be far away, and it can leave the reward range [1, 5].
Self-normalized IS divides by the observed total weight. It stays between the observed rewards, but a ratio of random sums is generally biased at finite sample sizes. Both weighted estimators converge to the target here. The naive average converges to the proposal’s average, which differs from the target unless the mixes match in this toy model. Owen, Monte Carlo theory, methods and examples, chapter 9.
Support matters: require q(x) > 0 wherever p(x) > 0 for this comparison. An outcome that is never sampled cannot be recovered by giving existing samples bigger weights. A tiny q where p is substantial creates large, rare contributions and may increase variance.
ESS = (Σᵢ wᵢ)² / Σᵢ wᵢ²
Weight effective sample size (ESS) summarizes how uneven the weights are. Here it lies from 1 to n when n > 0. It is a diagnostic, not an exact count of independent target samples or a guarantee of small error. It does not account for which rewards receive the weights.
- Xᵢ · one random outcome
- A categorical scalar in {A, B}, drawn independently from q. After drawing it, its table row is observed. Code compares one uniform random number with q(A).
- p, q · two distributions
- Two fixed length-2 probability vectors. Each entry is positive and each vector sums to 1. Inputs set the A entries; code computes each B entry as 1 minus A.
- wᵢ · one importance ratio
- A positive scalar p(Xᵢ)/q(Xᵢ); it is a weight, not a probability, and can exceed 1. Random before Xᵢ is drawn, known afterward. Code looks up the two probabilities and divides.
- rᵢ · one reward
- A scalar: 1 for A or 5 for B. Fixed given the outcome, random before sampling. Its weighted contribution is wᵢrᵢ.
- n, Σw, Σw² · running totals
- A nonnegative integer count and two scalar sums. Code accumulates every draw, even after its trace row or checkpoint is no longer displayed. Estimates are undefined until n > 0.
A small side example: entropy measures spread
For two outcomes, entropy measures uncertainty in which outcome appears. A 50/50 mix has the highest entropy. A more lopsided mix is more predictable. Reward size does not enter this calculation.
H(p) = −Σₓ p(x) ln p(x)
Target entropy: 0.5004 nats. Proposal entropy: 0.5004 nats. Maximum for two outcomes: ln 2 ≈ 0.6931 nats.
H is one deterministic scalar for a fixed categorical probability vector. Natural logarithms give units of nats; base-2 logarithms would give bits. For two outcomes, 0 ≤ H ≤ ln 2, using 0 ln 0 = 0 at the endpoints. The editable probabilities here stay away from 0 and 1.
The connection to reinforcement learning
In off-policy learning, a behavior policy b generates actions, while a target policy π is the one we want to evaluate. For an observed state and action, the analogous ratio is π(a | s) / b(a | s). A whole trajectory can require a product of these ratios when the initial-state distribution and environment dynamics match. Products can make variance grow sharply; matching support is still necessary.
PPO’s clipped surrogate is a different use of a probability ratio. It changes the optimization objective to limit incentives for some large policy changes. Clipping a ratio does not provide an unbiased importance correction. The sampling experiment here uses ordinary, unclipped importance weights. Schulman et al., Proximal Policy Optimization Algorithms.
Ready. Draw samples from q and compare the three estimates with the exact target average.
TRY THIS CONCEPT · Entropy
Move the mix. Build the uncertainty score.
Imagine an A/B spinner. A likely result is less surprising; a rare result is more surprising. Entropy averages that surprise using how often each result occurs.
The outcome colors label A and B; the probability and equation labels retain the lesson's reading key. The natural logarithm measures surprise in nats. A probability-zero outcome never occurs: its contribution is defined as 0 even though −ln(0) is not finite.
KL divergence asks a different question: how much did the distribution change?
Entropy describes one spinner’s uncertainty. KL divergence compares two spinners over the same outcomes. Imagine a strategy changing from an equal mix to an 80/20 mix. For each outcome, compare its old and new chance with a log-ratio, then average using the old chances.
Keep the outcome labels fixed while comparing the mixes
Old strategy p
New strategy q
0.5 × ln(0.5 / 0.8)
≈ −0.2350
0.5 × ln(0.5 / 0.2)
≈ +0.4581
Add → KL(old ∥ new) ≈ 0.2231 nats
Equation anatomy · the first distribution supplies the averaging weights · show / hide
| Symbols and jobs | Type, randomness, range | Example |
|---|---|---|
| \({\color{#006e78}{p}},{\color{#006e78}{q}},{\color{#006e78}{p}}_{{\color{#525968}{i}}},{\color{#006e78}{q}}_{{\color{#525968}{i}}}\) | Two fixed probability vectors and their entries; nonnegative, each vector sums to 1. Their roles here are old/new strategies. | p = [0.5,0.5]; q = [0.8,0.2] |
| \(D_{\mathrm{KL}},{\color{#525968}{K}},{\color{#525968}{i}},\sum,\log,\Vert\) | Deterministic nonnegative scalar, possibly infinite; positive category count; index 1…K; sum and natural log. The double bar separates distributions and is not a norm here. | KL(p ∥ q) ≈ 0.2231; KL(q ∥ p) ≈ 0.1927 nats |
A zero-p outcome contributes zero. If p gives an outcome positive chance but q gives it zero, this direction of KL is infinite. KL is not a symmetric distance. Two nearly certain strategies choosing opposite actions can both have low entropy while differing greatly. In policy optimization, a KL penalty, constraint or stopping check can discourage large changes; the states and averaging distribution must also be specified. PPO’s clipping alone is not a hard KL constraint. Probability and information-theory reference.
Where this lesson belongs: sampling, importance sampling, and entropy live in the general Math/ML toolbox. Complete returns, behavior versus target policies, and PPO are linked RL applications. Return to the method map to connect them. For RL reference reading, see Sutton and Barto, chapters 5–6.
Be First to Comment