Most Training Failures Are Setup Failures

Most training failures are setup failures. Bad input scale, bad initialization, missing normalization, careless regularization, and a mismatched loss make a good architecture look broken. A mismatch among those five wastes days that look like architecture problems.

Preprocessing removes friction that has nothing to do with the model

Mean subtraction centers the data. Scaling keeps feature ranges comparable. For images that usually means subtracting the training-set mean image, or the per-channel means.

Optimization works better when features sit on comparable scales and activations stop getting pushed into bad regions. Preprocessing removes variation the label never depended on, and it belongs before every other decision in the setup.

Initialization gives learning a fair start and nothing more

Weights too small and the signal vanishes. Weights too large and the activations or the gradients explode. Symmetric initialization makes every neuron in a layer learn the same thing. Random initialization breaks that symmetry, and it works only when the scale matches the layer.

Read the activation scale after the first forward pass. A collapse toward zero or a blowup names the initialization scale, before you touch the architecture.

Batch normalization and dropout change the function you evaluate

Batch normalization normalizes intermediate activations and learns a scale and a shift. It makes the network less sensitive to initialization and it can allow higher learning rates. It also changes the training system.

The layer uses batch statistics during training and stored running statistics during evaluation. You must track those running statistics and know what the layer does in each mode. Do not paste it everywhere.

Dropout disables units at random during training. The network cannot lean on one exact path, so it learns distributed representations that tend to generalize better. At test time dropout uses the full network with the correct scaling convention.

Forget the train() and eval() switch and you change the function you measure. The curve still moves and the number still prints, and both describe a model you are not shipping. I check the mode switch before I read any validation number.

Train-time and test-time paths must agree except where you meant them to differ

Preprocessing is part of the model contract. Subtract the training mean during training and you must subtract the same mean at test time. When the two input pipelines diverge, you evaluate the model in a different world than the one it learned.

The classification note put the trust boundary at the split. Leakage sends training information across that line. A mismatched pipeline sends the model across it. Both break the one instrument that reports the truth.

The only deliberate mismatch is training-only randomness such as augmentation or dropout.

Regularization is a budget on how much you trust the data

Every training setup decides how much to trust the data, the model, and the objective. Regularization is the explicit budget on that trust. It says how much freedom the model gets to fit the training data.

• L2 penalizes large weights, so the decision stops resting on a few fragile ones.
• Dropout removes units during training, so no single narrow path through the network carries the answer.
• Augmentation broadens the examples, so accidental correlations stop holding across the training set.
• Early stopping cuts the run before memorization dominates.

Each one answers the same question. How do we stop the model before it turns accidents into rules?

Training loss going down is a fact about the training set. The train-validation gap is the number to read. When training keeps improving and validation gets worse, the model is memorizing details that do not travel. When validation improves as training becomes a little harder, regularization is doing its job.

The loss is the game the model plays

The loss must match the task. Classification usually uses softmax cross-entropy or an SVM-style loss. Multilabel tasks need different treatment, and regression needs different objectives again.

Softmax cross-entropy and the SVM loss are the two the linear-classifier note weighed. Pick one and keep one name for it through the unit. A model tuned against a loss that states the wrong task moves numbers without moving the result.

The loss defines what counts as wrong, and every other control is tuned against that definition. Together they decide whether training learns structure or exploits convenience.

The Builder Test

Take a run you already have and plot the train and validation curves on the same axes. Then change exactly one thing: the preprocessing, the initialization scale, the regularizer, or the loss. Run it again and compare the gap.

An intervention that shrinks the gap while validation keeps improving is the one to keep. If the gap shrinks because both curves flattened, the change destroyed useful signal. Change one thing per run. Two changes and you cannot say which one moved the gap.

What Carries

Carry the gap. Generalization lives in the space between the examples you fit and the pattern you learn.

With the setup honest, the remaining question is how to run the training loop without lying to yourself. Keep the two curves on the screen while you answer it.