Neural Networks Are Learned Feature Machines

A hidden layer is a learned change of coordinates. Its only defense is that it makes the final comparison easier.

The linear classifier note ended on one averaged template per class. That template holds every pose and every background of a class in one picture, so it averages them. A hidden layer removes that specific limit.

It rewrites the input into coordinates where a straight boundary can do the work. The overview asked which representation survives an image nobody arranged for the model. This is the first stage where the model chooses that representation instead of receiving it.

Remove the nonlinearity and the stack collapses into one linear map

A neuron takes its inputs, computes a weighted sum, adds a bias, and sends the result through a nonlinearity. Take the nonlinearity out and two stacked layers multiply into one matrix. The depth stays on the page, and the model it defines is the template you already had.

That nonlinearity is the activation, and choosing it is a structural decision. ReLU, sigmoid, tanh, and their variants set how information and gradients move through the stack.

Depth is worth paying for when later layers reuse earlier features

Early layers pick up simple patterns. Later layers combine them into edges, textures, parts, and structure that belongs to a class. Each stage buys its inputs from the stage below, and only the first one reads pixels.

One averaged template cannot do this. It smears every mode of a class into one average. A stack keeps the modes apart, because a part detector can fire on two images that do not look alike.

Depth pays out only when the optimizer can train it and the data can support it. Otherwise the extra layers become capacity for memorization, instability, or shortcuts.

Activation choice is a decision about gradient flow

Sigmoid and tanh saturate. Once a unit sits in the flat region, the gradient it passes down is small. The layers under it learn slowly. ReLU is simple and often trains better, and it fails in its own way.

A unit that parks on the negative side passes no gradient, and it stays parked. That unit is finished learning for the rest of the run.

Universal approximation does not promise that SGD finds the function

Depth, width, and activation each set capacity, optimization difficulty, memory, speed, and overfitting risk. Universal approximation is a statement about capacity. It says a network can represent a wide family of functions.

It says nothing about whether SGD finds that function, whether the result generalizes, or what the training costs. Representable and trainable are separate properties, and generalizable is a third.

The useful question is whether this is the right amount of power for this evidence.

The Builder Test

Ask what changed between input space and hidden space. Take the examples the model gets wrong and find where they sit in the hidden representation. If they are still mixed in with the examples it gets right, the layer has not started doing work.

Then check which failures still look linear. Those are the cases the new coordinates left where they were.

The common failure is counting layers without asking what each one makes easier. When the data, the loss, or the optimization is wrong, more layers make the bug harder to see.

What Carries

Architecture is a bet about what structure the data has. A hidden layer is the smallest version of that bet. You claim that useful intermediate features exist in this data, and the training run settles it.

The bet raises the price of every setup choice. When the representation itself is learned, a bad setup looks exactly like a bad model. Audit the setup before you add another layer.