Math Toolbox · One focused lesson
Use a circle and a square to build Monte Carlo estimation before applying it to complete journeys.
Reading tips and color key
Start with the explanation and experiment. Open the symbol cards when you want the details. Teal: probabilities; blue: states; orange: actions; green: rewards; purple: values; gray: parameters and settings; crimson: losses and errors. Written labels give every symbol its role.
Estimate π with a circle and a square
Why use it? Estimate a quantity whose exact calculation is inconvenient, using outcomes you can sample.
How does it work? Drop points uniformly, mark each hit as 1 or 0, and average. Geometry supplies the factor of four.
What does it give you? A random estimate of π, with uncertainty that usually shrinks as the sample count grows.
When does it struggle? It is inefficient for computing π precisely. It is useful here because the geometry makes sampling error visible; more samples do not improve every individual run.
The circle’s diameter equals the square’s side length. Scatter independent points uniformly across the square: equal-area regions have equal chances. The fraction inside the circle estimates its share of the square’s area.
- Measure the shapes.Square: D × D = D²Circle: π × (D / 2)² = πD² / 4
- Divide the areas.(πD² / 4) ÷ D² = π / 4The common D² cancels. Size does not change the fraction.
- Replace area with counts.Inside fraction C / N ≈ π / 4Multiply by 4 → π̂ = 4C / N
Equation anatomy · area becomes a counting problem · show / hide
| Symbols and jobs | Type, randomness, range | Example |
|---|---|---|
| \({\color{#525968}{D}}\): diameter and side | Fixed positive scalar length | diameter = 2 |
| \(\pi,\widehat\pi\): constant and estimate | π is fixed; scalar estimate varies across samples, from 0 to 4 | 785 inside / 1,000 total → 3.14 |
| \({\color{#525968}{C}},{\color{#525968}{N}}\): inside and total counts | Integers; C random before sampling; chosen N positive; C from 0 to N | 4*inside/total |
This π is the geometry constant. Later, the same Greek letter names an RL policy; its written context tells you which meaning applies.
Math you can see · Sampling
Let random samples do the counting
First, estimate a circle’s area by scattering dots. Then see how to reuse samples drawn from a different distribution. Every dot, table row, and curve below comes from the experiment you run.
Equation key: probabilities · rewards · counts and settings · estimates. Chart colors identify the compared quantities and have a separate legend.
01 · Estimate by counting
How many dots land inside the circle?
A circle fits exactly inside a square. Drop points uniformly across the square: every equal-sized patch gets the same chance. The fraction inside estimates the circle’s share of the square.
Play adds 100 samples at a time and stops at the limit (at most 50,000). Pause keeps your progress. Reset applies the seed; changing D resets this experiment. Identical seeds reproduce identical points, regardless of batch size.
InsideOutside
Estimate π̂True π ≈ 3.14159
95% Wilson interval, scaled to π: —
This uses the inside/outside count as binomial data and multiplies its hit-rate interval by 4. The nominal 95% level describes approximate coverage over repeated experiments at a fixed sample count. It is not a guarantee for one run, and repeatedly checking then choosing when to stop does not preserve that coverage.
Inspect the latest 8 actual dots
| Sample | x | y | Inside? |
|---|---|---|---|
| Add dots to see their coordinates. | |||
Open the math and its anatomy
The square’s area is D². The circle’s radius is D/2, so its area is π(D/2)². Dividing gives P(inside) = π/4. Estimate that probability by c/n, then multiply by 4.
x, y ∼ Uniform(−D/2, D/2), independently
inside ⇔ x² + y² ≤ (D/2)²
π̂ = 4c/n
- D · one positive setting
- A fixed scalar length. Here 0.1 ≤ D ≤ 100. It sets both the square’s side and the circle’s diameter. Scaling both shapes does not change their area ratio.
- x, y · two sampled coordinates
- Two independent random scalars, together a length-2 vector. Each lies between −D/2 and D/2. Code draws two uniforms, shifts and scales them, then checks the squared distance.
- c, n · observed counts
- Nonnegative integer scalars; 0 ≤ c ≤ n. Increment n for every point and c only for an inside point. All points count even when older dots disappear from the display.
- π̂ · a random estimate
- A scalar in [0, 4] when n > 0. It varies across experiments and is fixed once those points are observed. Code computes 4 × c / n; no estimate is shown at n = 0.
More samples usually reduce Monte Carlo noise, but an individual next dot can move the estimate farther from π. Typical error scales like 1/√n, so roughly four times as many independent samples are needed to halve it.
Interval reference: NIST’s discussion of Wilson proportion intervals.
Ready. Add one dot to follow one sample, or 100 to see an estimate form.
02 · Correct for where samples came from
What if we sampled from the wrong mix?
Outcome A pays 1 point; outcome B pays 5 points. We want the average under the target distribution p, but our samples come from the proposal distribution q. Give each observation weight p/q to account for the mismatch.
| Outcome | Target p | Proposal q | Weight p/q | Reward |
|---|
Target average: 0.80 × 1 + 0.20 × 5 = 1.800 points. Proposal average: 4.200 points.
All three estimators use the same samples from q. Changing either probability resets this experiment using the current seed. Both outcomes keep positive probability, so sampling can reach everything the target needs.
Samples: 0
Weight ESS: —
Sum of weights: 0
Ordinary ISSelf-normalized ISNaive averageExact target average
Inspect the latest 8 actual samples
| Sample | Outcome | p | q | w = p/q | Reward | Contribution |
|---|---|---|---|---|---|---|
| Draw a sample to inspect its weight. | ||||||
Why the three estimates behave differently
wᵢ = p(Xᵢ) / q(Xᵢ)
Ordinary IS = Σᵢ wᵢrᵢ / n
Self-normalized IS = Σᵢ wᵢrᵢ / Σᵢ wᵢ
Naive average = Σᵢ rᵢ / n
Ordinary IS is unbiased for the target mean in this finite model: across repeated fixed-size experiments, its mean is exactly the target. A particular run can still be far away, and it can leave the reward range [1, 5].
Self-normalized IS divides by the observed total weight. It stays between the observed rewards, but a ratio of random sums is generally biased at finite sample sizes. Both weighted estimators converge to the target here. The naive average converges to the proposal’s average, which differs from the target unless the mixes match in this toy model. Owen, Monte Carlo theory, methods and examples, chapter 9.
Support matters: require q(x) > 0 wherever p(x) > 0 for this comparison. An outcome that is never sampled cannot be recovered by giving existing samples bigger weights. A tiny q where p is substantial creates large, rare contributions and may increase variance.
ESS = (Σᵢ wᵢ)² / Σᵢ wᵢ²
Weight effective sample size (ESS) summarizes how uneven the weights are. Here it lies from 1 to n when n > 0. It is a diagnostic, not an exact count of independent target samples or a guarantee of small error. It does not account for which rewards receive the weights.
- Xᵢ · one random outcome
- A categorical scalar in {A, B}, drawn independently from q. After drawing it, its table row is observed. Code compares one uniform random number with q(A).
- p, q · two distributions
- Two fixed length-2 probability vectors. Each entry is positive and each vector sums to 1. Inputs set the A entries; code computes each B entry as 1 minus A.
- wᵢ · one importance ratio
- A positive scalar p(Xᵢ)/q(Xᵢ); it is a weight, not a probability, and can exceed 1. Random before Xᵢ is drawn, known afterward. Code looks up the two probabilities and divides.
- rᵢ · one reward
- A scalar: 1 for A or 5 for B. Fixed given the outcome, random before sampling. Its weighted contribution is wᵢrᵢ.
- n, Σw, Σw² · running totals
- A nonnegative integer count and two scalar sums. Code accumulates every draw, even after its trace row or checkpoint is no longer displayed. Estimates are undefined until n > 0.
A small side example: entropy measures spread
For two outcomes, entropy measures uncertainty in which outcome appears. A 50/50 mix has the highest entropy. A more lopsided mix is more predictable. Reward size does not enter this calculation.
H(p) = −Σₓ p(x) ln p(x)
Target entropy: 0.5004 nats. Proposal entropy: 0.5004 nats. Maximum for two outcomes: ln 2 ≈ 0.6931 nats.
H is one deterministic scalar for a fixed categorical probability vector. Natural logarithms give units of nats; base-2 logarithms would give bits. For two outcomes, 0 ≤ H ≤ ln 2, using 0 ln 0 = 0 at the endpoints. The editable probabilities here stay away from 0 and 1.
The connection to reinforcement learning
In off-policy learning, a behavior policy b generates actions, while a target policy π is the one we want to evaluate. For an observed state and action, the analogous ratio is π(a | s) / b(a | s). A whole trajectory can require a product of these ratios when the initial-state distribution and environment dynamics match. Products can make variance grow sharply; matching support is still necessary.
PPO’s clipped surrogate is a different use of a probability ratio. It changes the optimization objective to limit incentives for some large policy changes. Clipping a ratio does not provide an unbiased importance correction. The sampling experiment here uses ordinary, unclipped importance weights. Schulman et al., Proximal Policy Optimization Algorithms.
Ready. Draw samples from q and compare the three estimates with the exact target average.
TRY THIS CONCEPT · Entropy
Move the mix. Build the uncertainty score.
Imagine an A/B spinner. A likely result is less surprising; a rare result is more surprising. Entropy averages that surprise using how often each result occurs.
The outcome colors label A and B; the probability and equation labels retain the lesson's reading key. The natural logarithm measures surprise in nats. A probability-zero outcome never occurs: its contribution is defined as 0 even though −ln(0) is not finite.
Monte Carlo means estimating through samples
An expectation is the probability-weighted average over possibilities. A sample average estimates it. The circle experiment averages “inside or outside” outcomes, then multiplies by four.
A point becomes a number
- Point 11
- Point 21
- Point 30
- Point 41
- Point 51
- Point 60
- Point 71
- Point 81
Equation anatomy · independent, identically distributed samples with finite variance · show / hide
| Symbols and jobs | Type, randomness, range | Example |
|---|---|---|
| \(X,x_{{\color{#525968}{i}}},f\): random input, recorded input, measurement function | Input shape depends on task; f is fixed with real scalar output | Point coordinates → 1 inside, 0 outside |
| \(\mu,\widehat\mu\): exact and estimated average | Scalars; μ fixed; estimate random across datasets | Average of 2, 4, 6 is 4 |
| \(\sigma,\operatorname{SE}\): standard deviation of f(X) and of its sample mean | Nonnegative scalars; fixed for a distribution and N | σ = 2, N = 100 → SE = 0.2 |
| \({\color{#525968}{N}},{\color{#525968}{i}},\sum,\mathbb{E}\): count, index, sum, expectation | Positive count; index 1…N; deterministic operators | sum(measurements)/N |
Variance measures squared spread; standard deviation is its square root. Four times as many independent samples halve the standard-error scale. An individual estimate can still get worse when another sample arrives. Correlated samples and heavy tails require more care.
In Monte Carlo RL prediction, the measurement is a completed episode’s return. Average returns after visits to a state to estimate its value. General Monte Carlo need not involve episodes; that is the RL application. See MC versus TD.
Be First to Comment