Every Equation Tells a Story

Reinforcement Learning · One focused lesson

Use this compact reference to choose a question. Each linked lesson explains one part of the story.

Reading tips and color key

Start with the explanation and experiment. Open the symbol cards when you want the details. Teal: probabilities; blue: states; orange: actions; green: rewards; purple: values; gray: parameters and settings; crimson: losses and errors. Written labels give every symbol its role.

Mystery Buttons and Changing Beliefs — Start with one repeated choice. Learn why exploration matters, then try Bayesian evidence and Thompson sampling.
Chance, Averages, and Beliefs — A probability describes a chance; an expectation combines chances; a posterior updates a belief. Keep those three jobs separate.
From One Step to a Whole Journey — First follow probability through a world. Then add choices and count the rewards of a journey.
A Future Worth Counting — What is a situation worth under a strategy? Build the answer from the next reward and the value of what follows.
Planning with a Rulebook — When transition chances and rewards are known, use the rulebook to improve a plan.
Learn Now or Wait Until the End — Monte Carlo waits for a completed journey. Temporal-difference learning starts from one transition.
Two Targets, Two Routes — The same transition can produce two different learning targets. Compare the next action each one assumes.
Random Dots and Fair Comparisons — Use a circle and a square to build Monte Carlo estimation before applying it to complete journeys.
Reweighting What We Observe — A sample’s contribution should reflect both what it measured and how likely we were to collect it.
Better Than Expected — Compare an action with the usual result, then see why policy updates use probability ratios.
Uncertainty and Changing Strategies — Entropy measures spread within a distribution. KL divergence compares two distributions in a specified direction.
Small Steps Down a Hill — Build one gradient step, then compare full-data and sampled gradients on the same known objective.
A Little Correction at a Time — An old average plus a fraction of its error can equal a freshly recomputed average. This pattern connects several learning methods.
When Downhill Leads to Different Valleys — Change the starting point while keeping the same loss. Watch actual gradient updates reach different valleys.
When Learning Signals Shrink or Explode — Multiply local derivatives one stage at a time. See why clipping a large gradient and restoring a small one are different problems.

A small language guide

Every symbol has a job. A scalar is one number; a vector is an ordered list; a matrix is a table; a tensor is an array that can have more axes. A function takes an input and returns an output. A probability is one number between zero and one; a distribution describes the chances of all possible outcomes.

A random variable represents an outcome before it is known. Uppercase \({\color{#172337}{\color{#1d4ed8}{S}}_{\color{#525968}{t}}}\) means the state we might find at time \({\color{#172337}{\color{#525968}{t}}}\); lowercase \({\color{#172337}{\color{#1d4ed8}{s}}}\) means a particular state, such as tile 12. A function can be fixed even when its input is random. A learned table is fixed once stored, but a new training run can produce a different table.

A parameter belongs to a model, such as a button’s unknown success chance or a neural network’s learned weights. A hyperparameter controls a learning setup, such as the learning rate. An operator tells us what to do: add a list, take an average, or find a maximum.

Color is a reading aid, never the only label. Blue: states · Orange: actions · Green: rewards and returns · Purple: values and targets · Teal: probabilities and policies · Gray: settings and counts · Crimson: estimation errors. Symbols and written labels carry the meaning. Method colors in experiments identify algorithms, a separate visual key.

One equation family, several kinds of new information

New estimate = old estimate + step size × (target − old estimate). The subtraction tells us the direction and distance. The step size controls how much of that distance we move. A negative error lowers the prediction; a positive error raises it.

Recognize the shared structure · show / hide

Recognize the shared structure
Method	Prediction	Target in words	Where information comes from
Bandit average	Button’s mean reward	Newest reward	One observed button press
Monte Carlo	State’s value	Completed return	Actual remaining episode
TD(0)	State’s value	Reward + discounted next state estimate	One transition and an estimate
SARSA	State-action value	Reward + discounted value of next selected action	One transition and behavior’s next action
Q-learning	State-action value	Reward + discounted best next action value	One transition and a maximum over estimates

The bandit mean uses a shrinking step of one divided by its visit count. Other updates may use a fixed learning rate or a schedule. Changing the target, the step schedule, or action selection changes the method’s behavior.

Reference route

A method can belong to several families

Ask four separate questions: Does it use a model? What is its target? Does it learn values, a policy, or both? Whose policy supplies the data? These labels describe different axes, not competing boxes.

Compare the forms taught here; variants can have different labels · show / hide

Compare the forms taught here; variants can have different labels
Method	Environment model?	Backup/target	Learns	Policy relationship
DP evaluation	Uses transition and reward model	Expected one-step backup; bootstraps	Values for a fixed policy	Specified policy; no behavior samples required
Value iteration	Uses model	Best expected backup; bootstraps	Optimal values, then greedy policy	Optimizes a policy through planning
MC prediction	No model required	Full sampled return; no bootstrap	Values	On-policy form here; off-policy variants exist
TD(0) prediction	No model required	Sampled reward + next estimate; bootstraps	Values	On-policy form here; variants exist
SARSA	No model required	Next chosen action value; bootstraps	Action values; policy through selection	On-policy
Q-learning	No model required	Maximum next action value; bootstraps	Action values; policy through selection	Off-policy greedy target

Prediction evaluates a strategy. Control improves one. Model-free means a transition/reward model is not needed for the update; it does not mean the algorithm has no tables, neural networks, or assumptions. A model-based agent may learn its model rather than receive it.

Read experiments carefully. A path shows one realization. Returns show consequences; success rate shows reliability; transition counts show experience used. Previous paths can be lighter, but keep method names and line styles visible. Planning with a known model and learning from sampled transitions have different information budgets.

Later topics and primary reading

The next learning modules · roadmap only

How the foundations grow into modern RL

The topics below are future modules, not fully taught by the equations above. Each starts with a limitation you can now recognize.

Extend the target

Multi-step learning → GAE

One step borrows an estimate early; a full return waits. Multi-step returns, TD(λ), and eligibility traces connect these choices. Generalized Advantage Estimation later combines TD errors to estimate how much better an action was than expected. GAE paper.

Grow beyond a table

Function approximation → DQN

Many states cannot fit in a table. Learn a value function with shared parameters. DQN combines neural action values with Q-learning, replay, and a target network. Its standard form handles a finite set of actions. Toolbox: vectors, tensors, loss, gradients, optimization. DQN paper.

Learn the chooser

Policy gradient → REINFORCE → actor–critic

Learn action probabilities directly. REINFORCE uses sampled returns to guide a policy gradient. Baselines reduce noise; an actor–critic adds a learned evaluator. Advantage compares an action’s value with the state’s usual value. Toolbox: logarithms, derivatives, gradients. REINFORCE paper.

Control policy changes

TRPO → PPO → GRPO

Large policy changes can undo useful behavior. TRPO uses a trust-region approach; PPO uses simpler surrogate objectives. GRPO, introduced in DeepSeekMath, uses group-relative reward information in place of a separate learned critic in its presented formulation. This is a related branch, not a universal ranking. TRPO · PPO · GRPO.

Math toolbox route: probability → conditional probability → Bayes → Thompson sampling. Another branch: random variables → expectation → Monte Carlo → value estimation. Later: derivatives → gradients → policy optimization → KL divergence and trust regions.

Before moving on, explain one sentence without symbols: why do SARSA and Q-learning produce different targets from the same transition? If you can name the next action each target assumes, you understand the relationship behind the formulas.

For a systematic reference, see Sutton and Barto, Reinforcement Learning: An Introduction, second edition, especially chapters 2–6 for bandits, MDPs, planning, Monte Carlo, and TD. Watkins and Dayan’s original Q-learning paper gives the tabular result under its stated sampling and learning-rate conditions. The stories and numerical exercises on this page are original teaching examples.

Continue: Mystery Buttons and Changing Beliefs

Continue: Chance, Averages, and Beliefs

Continue: From One Step to a Whole Journey

Continue: A Future Worth Counting

Continue: Planning with a Rulebook

Continue: Learn Now or Wait Until the End

Continue: Two Targets, Two Routes

Continue: Random Dots and Fair Comparisons

Continue: Reweighting What We Observe

Continue: Better Than Expected

Continue: Uncertainty and Changing Strategies

Continue: Small Steps Down a Hill