LEARN FROM EXPERIENCE
One world, several learning rules
An explorer wants to reach a goal. Every move changes its situation. Trying the same learning rule in an open field, a maze and a slippery world can produce very different journeys.
Start with “Learn a strategy” to compare Monte Carlo, SARSA and Q-learning. Then try “Understand a fixed strategy”: dynamic programming, Monte Carlo and TD estimate future reward while following the same random action rule. The lab explains the information each method gets and the work it performs.
Why compare these methods? A rule that works in open ground may behave differently beside a cliff. Keep the world and learning budget visible.
How? Choose a world and starting/goal tiles. Compare one episode or a bounded training run; expand the trace to inspect the target and update.
What should you measure? Return during exploration, goal-reaching rate, and experience used. A path alone is one realization.
Which method? Planning uses a model; MC waits while TD bootstraps; SARSA and Q-learning assume different next actions. Read the focused comparison before interpreting a winner.
THE GRID LAB · 6 × 6
Same world. Different ways to learn.
Help an explorer get from S to G. Choose the world, place the endpoints, and compare what two learning rules actually do.
Changing a learning setting starts a fresh experiment. The same seed and settings reproduce a run. α controls TD updates; Monte Carlo uses an average. View options and the run limit preserve learning.
Default: 200. Hard maximum: 5,000 per experiment. All run buttons obey this total limit. Control episodes also stop sampling after 100 moves; prediction has a 10,000-move safety guard.
The paths include exploration and slips. Optional grid numbers show the current largest Q(s, a) in each square. These current values do not rewind during path replay; the trace below shows the actual historical updates.
Reward while learning
Average score over the latest 20 training episodes (fewer at the beginning). Higher is better. Scores add rewards without discounting.
Look inside one square · current action estimates
Choose a square on either grid to see its four current action estimates. A larger Q estimate means a better predicted discounted return.
What makes this comparison fair—and what does it leave out?
Both methods get the same world, starting square, reward rules, settings and number of episodes. They receive independent random streams derived from the seed; the trips differ because the strategies differ. The table also reports environment steps: equal episodes do not mean equal experience.
For responsiveness, training trips stop after 100 moves. This is a sampling cutoff, not a terminal square. SARSA and Q-learning still bootstrap on the last move. Monte Carlo skips incomplete trips; that can bias its average toward shorter trips. The skipped count is shown. Try several seeds. This small experiment does not establish a universal winner.
All Q estimates start at zero. Greedy ties are broken randomly. ε stays constant, and terminal future value is zero. Monte Carlo uses first visits to state–action pairs; SARSA follows the next sampled action; Q-learning uses the largest next Q estimate.
What changes in the classic cliff comparison?
This compact 6 × 6 adaptation uses the cliff rules from Sutton and Barto, Example 6.6: ordinary moves receive −1, and cliff entry receives −100 followed by an immediate reset to S. Only G ends the episode. The setup button selects SARSA and Q-learning with γ = 1, α = 0.5 and ε = 0.1. The original textbook grid is larger; this is not a reproduction of its published learning curves.
SARSA’s target includes the next exploratory action actually chosen. Q-learning’s target uses the best next action estimate. Near a cliff, these assumptions can favor different routes. Inspect several seeds and the rewards collected during training; a single path does not establish that one algorithm is always better.
Here the strategy never improves: choose ↑, →, ↓ or ← with equal probability. We are asking “How good is each square under this strategy?”
Every round makes one DP sweep and gives MC and TD the same completed random trip, starting at a randomly chosen nonterminal square. A model sweep and a sampled trip require different amounts of work.
MC and TD show the same sampled path. DP uses the model and has no sampled trip of its own. Random starting squares are used for prediction so every nonterminal state can be sampled.
How to read the error and work counts
“Value error” is root mean squared error over accessible nonterminal squares, compared with a converged evaluation of this exact random strategy. Smaller is closer to that reference. DP uses the supplied transition model; MC and TD only receive sampled transitions.
DP reports state backups. MC and TD report sampled steps. TD updates after each transition. First-visit MC waits for the trip to end and averages returns. A 10,000-step safety guard skips an exceptionally long incomplete trip for MC; any such skips are reported. α affects TD only. The tint on these boards is a value estimate, not a method ranking.
Replay an episode and inspect its updates
Replay does not run training again. The last 20 episodes keep up to 200 transitions each; longer prediction trips report omitted moves. Score history keeps at most 2,000 episode summaries.
Be First to Comment