Reinforcement Learning · One focused lesson
The same transition can produce two different learning targets. Compare the next action each one assumes.
Reading tips and color key
Start with the explanation and experiment. Open the symbol cards when you want the details. Teal: probabilities; blue: states; orange: actions; green: rewards; purple: values; gray: parameters and settings; crimson: losses and errors. Written labels give every symbol its role.
SARSA and Q-learning: the target makes the difference
Why use it? You want to improve the choices themselves, including their later consequences.
How does it work? SARSA uses the next action actually selected. Q-learning uses the best-scored next action. Both correct the same current Q entry.
What does it give you? SARSA follows the behavior policy’s future; Q-learning targets greedy future behavior.
When does it struggle? Continuing exploration near a cliff can make SARSA favor a detour. Q-learning can learn a short greedy route while collecting poor exploratory returns. This is a comparison of objectives and behavior, not a safety guarantee.
Two explorers both keep scores for moves. One asks, “What will happen after the next move my strategy actually chooses?” The other asks, “What if I chose the move with the best current score next time?” Their update shape matches; their next-step questions differ.
SARSA uses the next selected action. Its name follows the experience sequence: state, action, reward, state, action. Q-learning uses the largest estimated next action value, even if exploration chooses something else.
Equation anatomy · choose either target, then use the shared update · show / hide
| Symbol | Meaning and kind | Form | Random? | Possible values | In code |
|---|---|---|---|---|---|
| \({\color{#172337}{\color{#7542b0}{Q}},{\color{#7542b0}{Q}}({\color{#1d4ed8}{S}}_{\color{#525968}{t}},{\color{#9a4700}{A}}_{\color{#525968}{t}}),{\color{#7542b0}{Q}}({\color{#1d4ed8}{S}}_{{\color{#525968}{t}}+1},{\color{#9a4700}{a}})}\) | Estimate table; current entry; candidate next entry | State × action matrix; scalars | Stored table fixed for this update; inputs vary by trajectory | Real values | Q[s][a] |
| \({\color{#172337}{\color{#1d4ed8}{S}}_{\color{#525968}{t}},{\color{#1d4ed8}{S}}_{{\color{#525968}{t}}+1}}\); \({\color{#172337}{\color{#9a4700}{A}}_{\color{#525968}{t}},{\color{#9a4700}{A}}_{{\color{#525968}{t}}+1},{\color{#9a4700}{a}}}\) | Two states; chosen current/next actions; candidate action | State IDs and action labels | Trajectory quantities generally random; candidate a fixed | Valid states and actions | nextAction = choose(next) |
| \({\color{#172337}{\color{#087444}{R}}_{{\color{#525968}{t}}+1}}\); \({\color{#172337}{\color{#7542b0}{y}}_{\color{#525968}{t}}^{\mathrm{SARSA}},{\color{#7542b0}{y}}_{\color{#525968}{t}}^{\mathrm Q},{\color{#7542b0}{y}}_{\color{#525968}{t}}}\) | Next reward; two alternative targets; target chosen for update | Scalar samples/calculations | Random across transitions; known at update | Real reward/value units | target = reward + gamma*nextValue |
| \({\color{#172337}{\color{#525968}{\alpha}},{\color{#525968}{\gamma}},{\color{#525968}{t}}}\); \({\color{#172337}{\max,\leftarrow}}\) | Learning rate, discount, time; largest value; assignment | Settings/index; operators | Fixed | \({\color{#172337}{0\lt {\color{#525968}{\alpha}}\le1;\ 0\le{\color{#525968}{\gamma}}\lt 1;\ {\color{#525968}{t}}\ge0}}\) | Q[s][a] += alpha*(target-Q[s][a]) |
Try the same transition. Reward −1, discount 0.9, next selected action’s value 2, best next value 6. SARSA’s target is −1 + 0.9 × 2 = 0.8. Q-learning’s target is −1 + 0.9 × 6 = 4.4. A larger target does not prove a better method; they evaluate different continuations.
Shared arrival: reward −1, discount 0.9
SARSA: follow the selected move
Selected next action → Q = 2
−1 + 0.9 × 2 = 0.8
Q-learning: take the largest score
Best-scored next action → Q = 6
−1 + 0.9 × 6 = 4.4
Only the next-value box changes. Both methods put their target into the same “old + step × (target − old)” update. Exploration can select the value-2 action even while the value-6 action is available.
On-policy means learning about the policy producing the behavior, as in ordinary SARSA. Off-policy means the target policy can differ from behavior, as in Q-learning learning a greedy target while behaving exploratorily. “Off-policy” does not mean “no policy.”
Near a dangerous shortcut, SARSA can account for continuing exploratory mistakes, while Q-learning’s target assumes greedy future choices. Compare both the rewards collected while learning and separate evaluation of a fixed policy. Outcomes depend on the environment, settings, data, and time allowed.
Be First to Comment