Math Toolbox · One focused lesson
Multiply local derivatives one stage at a time. See why clipping a large gradient and restoring a small one are different problems.
Reading tips and color key
Start with the explanation and experiment. Open the symbol cards when you want the details. Teal: probabilities; blue: states; orange: actions; green: rewards; purple: values; gray: parameters and settings; crimson: losses and errors. Written labels give every symbol its role.
A learning signal can shrink or grow on the way back
Pass a message through ten volume knobs. If every knob halves it, the message nearly disappears. If every knob doubles it, it becomes enormous. Backpropagation also passes a signal through a chain of multipliers.
Why study this? Early layers can learn very slowly when gradients vanish, or make unstable updates when gradients explode.
How does it happen? The chain rule multiplies local derivatives. Repeated derivative magnitudes below one shrink a scalar signal; magnitudes above one amplify it. A negative derivative also reverses the sign.
What should we inspect? Gradient norms by layer, activation ranges, loss, and actual update sizes. The toy below isolates multiplication; a real network has many interacting paths.
Symbols for a scalar chain · show / hide
| Symbol | Meaning | Example |
|---|---|---|
| L, h₀,…,hₘ | Scalar loss and intermediate scalar activations | hⱼ=c·hⱼ₋₁ |
| m, j, ∏ | Positive layer count, layer index, multiply all factors | Ten factors of 0.5 give 0.5¹⁰≈0.000977 |
| dL/dhₘ, dhⱼ/dhⱼ₋₁ | Incoming loss sensitivity and each local derivative | Incoming 1, local derivative c |
Try two starting points on the same landscape
Crimson: actual loss. Blue circles: successive parameter settings. No tangent or connecting shortcut is drawn. Displayed numbers are rounded; updates use full precision. Hover or focus a point for its update number.
Show the recorded updates
At most 300 updates. Stops when |gradient|≤0.001, the update budget is reached, or a step leaves the displayed interval [−2,2]. A small gradient is not a global-optimum certificate.
| Step | Old w | Gradient | New w | New loss |
|---|
Build a backward signal, multiplier by multiplier
Start with incoming gradient 1. Each stage multiplies it by the same local derivative c. This is a scalar chain-rule example, not a neural-network training run.
Show every multiplication
| Stage | Incoming | Multiply | Outgoing |
|---|
Connect it to a parameter update. If α=0.1 and g≈0.000977, the change −αg is only −0.0000977. If g=1024, the same learning rate gives −102.4. A large learning rate and an exploding gradient can both make a large update; they are different causes.
Respond to the cause
Tools for keeping useful gradients · show / hide
| Tool | Why and how | What to watch |
|---|---|---|
| Suitable initialization | Scale initial weights for the layer width and activation, helping forward signals and backward gradients retain useful magnitudes. | It improves the starting condition; training can still become unstable. |
| Activation choice | Saturating sigmoid units have small derivatives; ReLU has derivative 1 on its positive side, avoiding that particular shrinkage there. | ReLU’s negative side has zero derivative; inactive units can stop learning. No activation cures every architecture. |
| Residual connections | For hₙₑₓₜ=h+F(h), the derivative is I+JF. The identity path gives an additional route for the signal. | It helps optimization, but does not guarantee bounded or nonzero gradients along every direction. |
| Normalization | Normalize selected activation statistics to make signal scales easier to manage. | Batch size, data distribution, and placement matter; normalization is not a replacement for inspecting gradients. |
| Gradient norm clipping | Scale a gradient whose norm exceeds a threshold τ; cap exceptional update-driving signals while retaining their direction. | It does not restore a vanished signal or fix a bad objective. Frequent clipping is a reason to inspect the underlying instability. |
Clipping limits magnitude, not the loss itself · show / hide
| Symbol | Job |
|---|---|
| g, gclip, ‖g‖ | Original gradient vector, rescaled gradient vector, Euclidean norm; in the scalar demo, the norm is absolute value |
| τ>0 | Chosen maximum norm. For g=[6,8], norm=10 and τ=5 gives [3,4]. A zero gradient stays zero. |
Real networks multiply Jacobian matrices, not just identical scalar factors. Products can shrink some directions and amplify others. Recurrent networks repeatedly apply such chains across time, making long-range credit assignment especially sensitive.
Be First to Comment