Backpropagation Is Blame Accounting

Backpropagation is the chain rule kept as bookkeeping. The forward pass computes values. The backward pass computes blame. Almost every bug in it is a wrong shape, a stale cache, or a missing batch average.

The calculus is one line. The work is keeping shapes, cached values, and gradient routes correct across many gates. Most implementation bugs are bookkeeping errors that still cost you a training run.

Every gate keeps a two-line contract

A network is a graph of small operations: add, multiply, max, matrix multiply, and a nonlinear activation. Each one stores the value it produced and owes its inputs a derivative on the way back. Those two facts are the whole contract.

The upstream gradient says how much the final loss moves when the output of this gate moves. The local gradient says how much that output moves when one input moves. Multiply them and the input has its share.

upstream = dL_dout
local = dout_dx
dL_dx = upstream * local

An operation sees its own local derivative and the signal arriving from the operation after it. Nothing else about the network reaches it. Chain those backward and every parameter gets a gradient.

The forward pass buys memory so the backward pass can exist

The backward pass runs on values the forward pass already computed. A multiply gate needs its original inputs. A ReLU needs the side of zero it landed on, and a softmax loss needs the probabilities it produced.

This is why a neural net library stores activations during the forward pass. The memory is deliberate. It buys the ability to compute gradients later.

Gradient flow has four patterns

Four patterns cover most of the gates you derive by hand. Each pattern comes with the value the gate must cache to run it.

• Add distributes the upstream gradient to both inputs, and caches nothing.
• Multiply sends each input the upstream gradient times the other input, so it caches both.
• Max routes the whole gradient through the winner, so it caches which input won.
• ReLU passes gradient for positive activations and blocks it for negatives, so it caches the sign.

Once you can name the pattern at each gate, the backward pass is local rules chained in reverse. The forward pass decides the routing, and the cache is where it wrote that decision down.

Local derivatives are what make autodiff modular

Give every operation one interface. The forward returns an output and stores what it needs. The backward receives the upstream gradient and returns gradients for its inputs.

Matrix multiply, ReLU, a normalization layer, and a loss all participate on that interface. Each one has to know only its local derivative. A new operation joins an autodiff system without that system knowing anything else about your model.

The library handles gradients by running this interface for you. You still have to know what it saves, routes, multiplies, and sums.

The bugs are boring, which is why they survive

Backprop bugs arrive as a wrong transpose, an omitted batch average, or a forgotten cache. They also arrive as a gradient with the wrong shape, or a regularization term applied to the bias. A wrong broadcast still produces numbers. The numbers land on the wrong weight.

Shape discipline keeps blame arriving at the right parameter. Assert the shape of every gradient against the tensor it corrects. The assertion fails where the bug is.

A gradient that is absent, too large, or exactly zero has a local cause. Look at the operation, the shape, the activation, or the loss connection.

Run a numerical gradient check on a tiny case before you trust a large training run. The optimization note ran that check on the whole loss. Here it points at one gate.

The Builder Test

Trace one scalar by hand. Write the forward values and the loss, then walk backward one operation at a time. Shrink the graph until you can inspect every local derivative by hand.

The trace itself is simple. The hard part is doing it the same way through many gates and keeping the thread. If the manual trace disagrees with the code, trust the trace until you find the bug.

What Carries

Learning is prediction, mistake, blame, correction. Backprop is the blame made precise.

The loss decides the goal, and you chose the loss back in the linear classifier note. Backprop carries the consequence of that goal backward through the network. With blame delivered to every parameter, the open question is what the middle parameters are supposed to represent.