Math Toolbox · One focused lesson
Build one gradient step, then compare full-data and sampled gradients on the same known objective.
Reading tips and color key
Start with the explanation and experiment. Open the symbol cards when you want the details. Teal: probabilities; blue: states; orange: actions; green: rewards; purple: values; gray: parameters and settings; crimson: losses and errors. Written labels give every symbol its role.
One dial, one visible bowl
Why use it? The loss is differentiable and you want a direction that improves it locally.
How does it work? Evaluate the slope, multiply by a learning rate, then subtract that displacement.
What does it give you? An updated parameter. On this bowl, the best position is 3 and small enough steps approach it.
When does it struggle? A large step can overshoot. On a non-convex objective, a small gradient does not prove that you found the best solution.
Our dial’s ideal position is 3. Squared distance from 3 gives a smooth, bowl-shaped error. The derivative tells us the local slope. Subtracting a small multiple of that slope moves downhill.
w − 3 = 0 − 3 = −3
Square it to get loss 9.
2 × (−3) = −6
The negative slope says a small move right lowers loss.
−0.25 × (−6) = +1.5
New position 0 + 1.5 = 1.5; new loss 2.25.
Equation anatomy · a completely known objective · show / hide
| Symbols and jobs | Type, randomness, range | Example |
|---|---|---|
| \({\color{#525968}{w}},{\color{#525968}{w}}_{{\color{#525968}{k}}},{\color{#525968}{w}}_{{\color{#525968}{k}}+1}\) | Real scalar parameter, current and next settings; deterministic here | w = 0 |
| \({\color{#ae2455}{f}},{\color{#ae2455}{f’}}\): objective and derivative | Fixed scalar functions; f nonnegative, derivative any real number | At w = 0: loss 9, slope −6 |
| \({\color{#525968}{\alpha}},{\color{#525968}{k}}\): learning rate and iteration | Chosen positive scalar; nonnegative integer index | w -= 0.25*gradient |
Try it. Start at 0 with learning rate 0.25. The slope is −6, so the next position is 0 − 0.25 × (−6) = 1.5. Loss falls from 9 to 2.25. Another step reaches 2.25, with loss 0.5625.
Steps can be too large. At learning rate 1, this example jumps between 0 and 6 without improving. At 1.1, the first jump reaches 6.6 and raises the loss to 12.96. Crossing the minimum is not always failure: a smaller overshoot can still reduce error. The safe step scale depends on the objective’s curvature.
More dials: derivatives become a gradient vector
A derivative measures sensitivity to one input. A partial derivative changes one parameter while holding the others fixed. Collect those partial derivatives into a vector: the gradient. For differentiable functions, it points toward steepest local increase under ordinary Euclidean distance; descent moves the opposite way.
Equation anatomy · a whole vector and the chain rule · show / hide
| Symbols and jobs | Type, randomness, range | Example |
|---|---|---|
| \({\color{#525968}{\mathbf{w}}},{\color{#525968}{w}}_{{\color{#525968}{j}}},{\color{#525968}{d}},{\color{#525968}{j}}\) | Real parameter vector, scalar component, positive dimension, index 1…d; fixed at evaluation | Two dials → vector [1,−2] |
| \({\color{#ae2455}{F}},\nabla {\color{#ae2455}{F}},\partial {\color{#ae2455}{F}}/\partial {\color{#525968}{w}}_{{\color{#525968}{j}}}\) | Scalar objective; d-entry gradient vector; one real scalar partial derivative | A vector is not one gradient component |
| \(g,h,g’,h’,d/d{\color{#525968}{w}},\partial,\nabla\) | Fixed differentiable real scalar functions, derivatives, and differentiation operators | g(w) = w−3; h(u) = u² → 2(w−3) × 1 |
The chain rule follows how one calculation feeds another. Backpropagation applies it efficiently through a computation graph. It calculates derivatives of the specified computation. Gradient reference.
Full batch and SGD can optimize the same known equation
Why use it? A dataset may be too large to reread for every update.
How does it work? Full batch averages all gradients; SGD samples one; a mini-batch averages a few.
What does it give you? A tradeoff between cost per update and gradient noise, for the same known objective.
When does it struggle? Full batch can be expensive. Small batches fluctuate, and a single update can raise full-data loss. Compare data processed as well as update count.
The lab uses five fixed target values: −1, 1, 3, 5, 7. Fit one number to them by minimizing mean squared error. Full-batch gradient descent averages every example’s gradient. Stochastic gradient descent (SGD) samples one example; minibatch SGD averages a small group.
2[−3 − (−1)]
−4
2(−3 − 1)
−8
2(−3 − 3)
−12
2(−3 − 5)
−16
2(−3 − 7)
−20
↓ Choose a set, add its slopes, then divide by how many you chose.
Full gradient · all five
(−4 − 8 − 12 − 16 − 20) ÷ 5 = −12
Use every target on each update.
SGD · one target
Example: sample y = 1.
−8 ÷ 1 = −8
A different sampled target gives a different slope.
Mini-batch · two targets
Example: sample y = −1 and 1.
(−4 − 8) ÷ 2 = −6
Average only those selected slopes.
All three then use new w = old w − α × chosen average slope. These sampled sets are worked examples; the interactive lab below records the sets actually drawn.
Equation anatomy · exact finite-data gradient versus a sampled estimate · show / hide
| Symbols and jobs | Type, randomness, range | Example |
|---|---|---|
| \({\color{#ae2455}{L}},{\color{#525968}{w}},{\color{#7542b0}{y}}_{{\color{#525968}{i}}}\) | Fixed scalar loss, real adjustable parameter, fixed real target; lab loss at least 8 | mean(targets.map(y => (w-y)**2)) |
| \({\color{#ae2455}{L’}},{\color{#ae2455}{\widehat g}}\) | Full gradient function and sampled gradient estimate; real scalars; estimate varies by batch | 2*mean(batch.map(y => w-y)) |
| \({\color{#525968}{N}},{\color{#525968}{i}},{\color{#525968}{B}},|{\color{#525968}{B}}|,\sum,\in\) | Data count 5; index 1…5; random nonempty subset; size; sum; membership | Batch sizes 1, 2, or 5; sampled without replacement |
The targets’ mean is 3 and their variance is 8. That adds a vertical shift of 8 to our first bowl, preserving its gradient and minimizing position. The lab’s minimum loss is 8, not zero.
Try the lab’s starting point. At w = −3, the full gradient is −12. A step of 0.15 moves to −1.2, lowering loss from 44 to 25.64. The five individual gradients are −4, −8, −12, −16, and −20. Their mean is −12; a sampled one can disagree substantially.
THE OPTIMIZATION LAB
Take a step. Watch the loss change.
Five targets are −1, 1, 3, 5, 7. Choose one number w that gets close to all five. The known average squared loss is L(w) = mean[(w − y)²] = (w − 3)² + 8. Its minimum is at w = 3. The extra 8 lifts the curve without changing its gradient.
First, understand the slope without machine learning
For f(w) = (w − 3)², the derivative is f′(w) = 2(w − 3). At w = −3 the slope is −12. With learning rate α = 0.15, the update is −3 − 0.15 × (−12) = −1.2: move right, toward the minimum. A negative slope means moving right initially lowers the function. A step that is too large can overshoot.
A gradient is the same idea for several parameters: one partial derivative per parameter. Backpropagation uses the chain rule to calculate gradients efficiently. SGD changes which data contribute to an update; it does not replace a known equation with an unknown one.
Change settings, then select Reset optimization to apply them and start a fresh experiment. Set the gradient threshold to 0 to use only the budget/safety stop. Here the full objective is small enough to monitor exactly; real training often uses validation checks and patience. A small single-sample gradient would not prove the full gradient is small.
Where did this step come from?
Follow the latest actual update from targets → slopes → a parameter move → loss. Purple numbers are target data, crimson numbers are gradients or loss, and gray numbers are parameters or settings. Method borders match the plot.
Numbers are rounded to three decimals; calculations use full precision. The full loss comes from the same cached diagnostic shown in the plot.
What each method knows and computes
Both know the same loss formula and target data. Full gradient descent averages all five individual gradients 2(w − y). SGD randomly picks one; mini-batch descent averages a random subset without replacement. The subset gradient is an unbiased estimate of the full gradient at the current w. With a constant learning rate, sampled updates can bounce around the minimum.
Batch size 5 uses all data, so both updates agree. Gradient-term counts report optimization work; evaluating the full loss and the full-gradient stopping check uses five additional data terms per update, shown separately. These counts begin after initialization and exclude drawing the chart. The cards, plot points and trace reuse cached diagnostic values. This tiny lab compares update behavior, not wall-clock performance.
Try α = 1 and threshold = 0: the full-gradient method oscillates. Try α > 1: it diverges until the budget or safety bound stops it. A finite budget is always enforced. For this quadratic and constant α, 0 < α < 1 makes full-gradient iterates approach the minimum.
In RL, a neural network still has a defined loss or policy objective. Experience supplies sampled training targets or gradient estimates. Some methods treat bootstrapped targets as fixed during an update (a semi-gradient). Sampling, differentiation and importance weighting have different jobs.
Show the numerical update trace
Latest 20 updates per method, newest first. Each row records the actual targets, gradient and result. At most 100 update records per method are retained.
| Method | Update | Sampled targets | w before | Gradient | w after | Full loss after |
|---|
Uniform sampling gives an unbiased gradient estimate at the current parameter. A batch containing all five examples exactly matches the full gradient. The objective equation is known in both methods. Sampling changes the work and noise per update. A stochastic environment introduces uncertainty when estimating a population objective too. Bottou’s SGD reference.
Three things that should not be confused
A gradient
The derivative vector of a specified objective at specified parameters. It is a mathematical object.
A sampled gradient estimate
An approximation formed from sampled examples or trajectories. Randomness concerns the estimate; its individual derivatives may be computed exactly.
A derivative-free search
A method using objective evaluations without analytic derivatives, such as trying candidate settings. Finite-difference slope estimates also use evaluations, but approximate derivatives numerically.
Gradient methods need useful derivatives, not necessarily a perfectly known population average. Derivative-free methods can work when derivatives are unavailable, at the cost of additional objective evaluations.
When should the dial stop moving?
Set a time or update budget. When affordable, monitor the full objective and gradient norm. The lab’s optional gradient threshold checks the full gradient for both methods. A full-objective check also costs work in real tasks; count monitoring separately from examples used for updates.
With noisy updates, use repeated measurements and patience before declaring a plateau. For data-driven learning, use a separate validation set to check whether lower training loss helps unseen examples.
A stopping rule is evidence, not proof of the best answer. A small sampled gradient can hide a large full gradient. Even a zero full gradient can occur at a saddle point or a poor local minimum. A plateau can mean a tiny learning rate, noisy measurements, or limited progress. Report why training stopped. Optimization and early-stopping reference.
Be First to Comment