Math Toolbox · One focused lesson
A probability describes a chance; an expectation combines chances; a posterior updates a belief. Keep those three jobs separate.
Reading tips and color key
Start with the explanation and experiment. Open the symbol cards when you want the details. Teal: probabilities; blue: states; orange: actions; green: rewards; purple: values; gray: parameters and settings; crimson: losses and errors. Written labels give every symbol its role.
02 · Math when it becomes useful
Chance, expectation, and Monte Carlo
A bag has one card worth 6 points and three cards worth 2 points. Draw, record the score, replace the card, and shuffle. You will never draw a 3-point card, but 3 is your expected score: the average you would approach over many independent draws.
Expectation weights each possible outcome by its chance. Monte Carlo estimation approximates that expectation by sampling outcomes and averaging them. It is a general mathematical tool, used beyond reinforcement learning.
Equation anatomy · exact average and sampled average · show / hide
| Symbol | Meaning and kind | Form | Random? | Possible values | In code |
|---|---|---|---|---|---|
| \({\color{#172337}{X,x,\mathcal{X}}}\) | Quantity, possible outcome, outcome set | Scalar random variable; scalar; set | \({\color{#172337}{X}}\) is random; \({\color{#172337}{x}}\) is a fixed candidate | Here \({\color{#172337}{\mathcal{X}=\{2,6\}}}\) | score = drawCard() |
| \({\color{#172337}{p(x)}}\) | Probability mass at outcome x | Scalar entry of a distribution | No for a fixed distribution | 0–1; all masses sum to 1 | probability[6] = 0.25 |
| \({\color{#172337}{\mathbb{E},\sum,\in}}\) | Expectation, addition, set membership | Operators/notation | No | Expectation returns a scalar here | 2*0.75 + 6*0.25 |
| \({\color{#172337}{x_{\color{#525968}{i}},\widehat\mu_{\color{#525968}{N}}}}\) | Recorded draw; estimated mean | Scalar; scalar statistic | Fixed after recording; random across new samples | Draws 2 or 6; mean 2–6 | sum(scores)/scores.length |
| \({\color{#172337}{\color{#525968}{N}},{\color{#525968}{i}}}\) | Sample count and index | Integers | Fixed here | \({\color{#172337}{\color{#525968}{N}}\ge1;\ {\color{#525968}{i}}=1,\ldots,{\color{#525968}{N}}}\) | N = 100 |
The exact average is 2 × 0.75 + 6 × 0.25 = 3. Four draws of 2, 2, 2, 6 also average to 3; four draws of 2, 2, 2, 2 average to 2. More samples usually stabilize an estimate, but each added sample does not guarantee improvement.
Variance measures spread around the mean; standard deviation expresses that spread in the original units. Under independent sampling with finite variance, typical averaging error shrinks in proportion to one over the square root of the sample count. Roughly four times as many samples halve that scale of error.
Conditional probability means a chance given information: the chance of a star given that we pressed button B. With discrete outcomes we assign probability to individual outcomes. With a continuous quantity, such as an unknown success chance, we use a density. Probability is area over an interval; the density’s height can exceed 1, and a single exact point normally has probability zero.
03 · Keep uncertainty visible
Bayes: change a belief when evidence arrives
Why use it? Keep uncertainty visible instead of treating one lucky result as truth.
How does it work? Multiply each hypothesis’s prior by the observed-data likelihood, then divide by the total weight.
What does it give you? A posterior distribution, not just a single best guess.
When does it struggle? The answer depends on the likelihood and prior. A precise posterior under a wrong model can still be misleading.
A mystery coin might be fair, or it might favor heads. Before flipping it, you give both possibilities equal weight. Then you see heads. That result fits both possibilities, but it fits the heads-favoring coin better. Your belief shifts; one flip does not prove the answer.
A prior is belief before these observations. A likelihood asks how compatible the observed data are with each hypothesis. A posterior is the updated belief. Multiply likelihood by prior, then normalize so the updated probabilities add to one.
Fair coin
Prior 0.50 × heads likelihood 0.50
Weight 0.25
Heads-favoring coin
Prior 0.50 × heads likelihood 0.80
Weight 0.40
↓ Add the two surviving weights: 0.25 + 0.40 = 0.65
Fair posterior
0.25 ÷ 0.65 = 5/13 ≈ 38.5%
Heads-favoring posterior
0.40 ÷ 0.65 = 8/13 ≈ 61.5%
The denominator is the same total for both coins. Division changes the scale so the two posterior probabilities sum to one; their ratio stays 5:8. This is exactly where the fraction below comes from.
Equation anatomy · discrete Bayes’ rule · show / hide
| Symbol | Meaning and kind | Form | Random? | Possible values | In code |
|---|---|---|---|---|---|
| \({\color{#172337}{H,h,h’,\mathcal{H}}}\) | Hypothesis variable; candidate labels; hypothesis set | Categorical variable; labels; finite set | \({\color{#172337}{H}}\) represents uncertainty; labels are fixed | Fair or biased here | hypotheses = ['fair','biased'] |
| \({\color{#172337}{D,d}}\) | Data variable and observed data | Outcome or list of outcomes | \({\color{#172337}{D}}\) before observation; \({\color{#172337}{d}}\) now fixed | Heads/tails in this example | observed = 'heads' |
| \({\color{#172337}{\color{#006e78}{P}}(H=h),{\color{#006e78}{P}}(H=h\mid D=d)}\) | Prior and posterior probabilities | Scalar entries in distributions over hypotheses | Deterministic given model and observed data | 0–1; each distribution sums to 1 | posterior[h] |
| \({\color{#172337}{\color{#006e78}{P}}(D=d\mid H=h)}\) | Likelihood of fixed data as hypothesis varies | Function of hypothesis; scalar output | Fixed for each candidate here | 0–1 for these discrete data | likelihood['biased'] = 0.8 |
| Denominator; \({\color{#172337}{\sum,\mid,\in}}\) | Evidence \({\color{#172337}{\color{#006e78}{P}}(D=d)}\); sum, given, membership | Scalar normalizer; notation | Fixed under the model | Evidence must be positive | weight[h]/sum(weights) |
Try the numbers. Prior probabilities: fair 0.5, biased 0.5. Heads chances: fair 0.5, biased 0.8. After one head, the weights are 0.25 and 0.40. Their sum is 0.65, so the biased coin’s posterior probability is 0.40/0.65 = 8/13, about 61.5%.
Likelihood is not posterior. The likelihoods 0.5 and 0.8 sum to 1.3. They are probabilities over possible data for each fixed hypothesis, not a probability distribution over the two hypotheses. Bayes’ rule supplies the prior weighting and normalization.
Beta–Bernoulli: a belief about a button’s success chance
A button returns one star or no star: a Bernoulli outcome. Its unknown success chance can lie anywhere from 0 to 1. A Beta distribution describes our uncertainty about that chance. With independent outcomes conditional on a fixed chance, a Beta prior updates neatly after counted successes and failures.
Equation anatomy · continuous belief, binary observations · show / hide
| Symbol | Meaning and kind | Form | Random? | Possible values | In code |
|---|---|---|---|---|---|
| \({\color{#172337}{\color{#006e78}{\theta}}}\) | Unknown success-probability parameter | Scalar | Fixed in the toy environment; Bayesian belief models uncertainty over it | 0–1 | sampleBeta(a,b) draws a plausible chance |
| \({\color{#172337}{\operatorname{Beta}}}\); \({\color{#172337}{\sim}}\) | Distribution family; “has this distribution” | Continuous distribution on an interval | Distribution is fixed given its parameters; draws vary | Support [0,1]; total density area 1 | {a:4,b:2} |
| \({\color{#172337}{\color{#525968}{a_0}},{\color{#525968}{b_0}},{\color{#525968}{c}},{\color{#525968}{f}}}\) | Prior shape parameters; success and failure counts | Four scalars | Prior chosen; counts fixed after observing data | \({\color{#172337}{\color{#525968}{a_0}},{\color{#525968}{b_0}}\gt 0;\ {\color{#525968}{c}},{\color{#525968}{f}}}\) nonnegative integers | a = 1 + successes |
| \({\color{#172337}{D,\mid,\mathbb{E}}}\) | Recorded outcomes; conditioning; expectation | Data sequence; operators/notation | Data fixed in this update; mean deterministic | Data contain 0/1; mean lies in [0,1] | mean = a/(a+b) |
Try the numbers. Start with Beta(1,1), a uniform prior. Three successes and one failure produce Beta(4,2). Its mean is 4/6 = 2/3. The observed success fraction is 3/4; the posterior mean also includes the prior.
Thompson sampling draws a possible success probability from each button’s posterior, chooses the largest, observes the actual 0/1 reward, and updates that button’s counts. A broad belief leaves room for exploration. A posterior is still conditional on its assumptions: a button whose behavior changes over time needs a model that can adapt. Primary reference: Russo and colleagues, Thompson sampling tutorial, §3.
Be First to Comment