Training Neural Nets Without Lying to Yourself

Training is an inspection loop. A network that cannot memorize ten examples has not earned a larger dataset.

The loop is cheap and the run is not. You examine gradients, overfit a tiny batch, and compare train behavior with validation behavior. Each reading decides whether the evidence supports the next move.

Overfit ten examples before you touch a real dataset

Take ten labeled examples and train until the model reproduces every one of them. A working pipeline memorizes ten examples without effort. If it cannot, the pipeline is broken.

The usual causes are wrong labels, gradients that never arrive, and regularization set too strong. A loss wired to the wrong tensor does the same, and so does broken data loading. Each survives a full run, which is the expensive way to find it.

The tiny-batch test is the gradient check from the backprop note, moved one level up the stack. That check tested one gate at a time. This one tests the whole pipeline against a task that is too easy to fail. Two more checks cost minutes: examine the augmentation, and make sure the metric means what you think it means.

Momentum and adaptive methods change how the update moves

Plain SGD zigzags through ravines. Momentum keeps a velocity vector that smooths the path and carries the update through consistent downhill directions. Nesterov momentum peeks ahead before it computes the gradient. RMSProp and Adam scale updates by recent gradient statistics.

Every one of these changes the shape of the curve the optimization note taught you to read. The data underneath stays exactly as wrong as it was, so validation still decides.

An optimizer cannot fix bad labels, broken preprocessing, or leakage

Adam, RMSProp, momentum, and learning-rate schedules make training smoother. They leave bad labels, broken preprocessing, impossible tasks, and leakage where they were. The setup note made preprocessing part of the model contract, and no optimizer renegotiates that contract.

When one optimizer is the only thing that makes a configuration work, inspect the configuration instead of tuning it. Treat the rescue as a symptom and go find what it covered.

Hyperparameter search needs a hypothesis, a budget, and a record

Search learning rate and regularization on log scales, because useful values differ by orders of magnitude. Run coarse sweeps first and keep them short. Refine only inside a region that already looks good.

The validation set that ranks those sweeps is the same split the classification note sealed. Every sweep spends a little of it, so keep the count. Record the settings you tried, what stayed fixed, and which images failed.

A sweep that runs until a number improves has produced a number and nothing else. Write the hypothesis before the run starts, and the result answers a question you can name.

Ensembles buy accuracy with complexity you must pay to serve

Different models make different mistakes, and that disagreement is where the accuracy comes from. You pay for it in training time, serving cost, and how hard the system is to explain. Take the trade when the gain is worth the cost, and name the cost first.

Change one knob at a time or the run teaches nothing

Two changes in one run produce one number and no way to attribute it. The run after that inherits the confusion and costs more to interpret.

A run with a broken validation split does not deserve a leaderboard. Scaling a confused experiment makes the confusion expensive, because longer runs hide wiring mistakes behind noise you paid for.

A new architecture does not rescue a broken training loop. Run the boring checks first, and the architecture becomes testable.

The Builder Test

Babysit the first run through four readings, in order.

• The first loss value, which rules out a loss wired to the wrong tensor.
• The tiny-batch overfit, which rules out the rest of the wiring.
• The gradient scale, which rules out dead gradients and an unstable initialization.
• The train-validation gap, which separates a model that learned structure from one that memorized.

A failed reading ends the run there. Fix that one thing, restart the four, and carry exactly one knob change into the next full run.

What Carries

The small run is the cheapest teacher you have. A big run teaches the same lesson slowly, and you pay for the delay in compute.

So shrink the next experiment until every part of it is visible. The smallest honest experiment has two dimensions, where nothing hides behind size.