At Two Dimensions Nothing Hides

A spiral in two dimensions is the cheapest place to watch a linear model fail and a hidden layer fix it. The data has two coordinates, so the decision boundary is a line you can draw and look at. The failure arrives as a shape instead of a cause you inferred.

The spiral defeats a straight boundary no matter how you tune it

The linear classifier maps each point to class scores and trains with softmax loss plus regularization. It draws straight boundaries and nothing else. The spiral classes wrap around one another, so no straight boundary separates them.

That failure is the reason to run the linear model first. A straight boundary losing to a curve names its own cause, so you stop looking at the learning rate and the data loader. The linear-classifier note priced this limit as one averaged template per class. In two dimensions the same limit is a straight line, and the spiral is built to beat it.

One hidden layer carves the space into regions

A hidden layer transforms the input before classification. ReLU units carve the space into regions, and the final linear classifier works on that learned feature space rather than the raw two coordinates. The visible result is a curved, piecewise boundary.

The boundary bends because the network learned a feature transformation. The model finds an intermediate space where the final decision is easier. At this size you watch that happen rather than accept it on report.

The toy contains the entire loop

Four ideas from earlier notes run at once in this toy. The score function turns points into class scores and the softmax loss says how strongly the model is wrong. Backpropagation delivers the blame to every parameter, and gradient descent spends it on an update.

• Initialize the weights with small random values so units learn different features.
• Run a forward pass to compute class scores.
• Compute the softmax loss and add regularization.
• Backpropagate the gradients through the score function.
• Update the weights with gradient descent.
• Monitor training and validation behavior, and trust no single number.

Once that loop is real in two dimensions, the larger image models stop being mysterious. They run bigger, more structured versions of the same loop.

Regularization moves the boundary where you can see it move

Turn the penalty down and the hidden-layer model overfits the toy data. A larger hidden size makes the boundary more flexible, and too little regularization makes it jagged around individual points. Capacity and penalty stop being quantities you infer and become a shape on the screen.

The training note makes the tiny-batch overfit test the first gate before scale. This is the same gate with the answer drawn. A hidden layer that cannot overfit a spiral with the penalty turned down has a broken wire. You find it before a real dataset is involved.

A small model you cannot explain is borrowed confidence

Every part is visible at this size: data, scores, loss, gradients, updates, and boundary. The spiral is unrealistic on purpose, because removing scale is what lets you inspect the mechanism.

Large models give confusion room to hide behind parameters. A confusion that survives the spiral only becomes more expensive on the next run. Explain the small case before you buy the large one.

The Builder Test

Run the linear model on the spiral first and confirm the limit yourself. Add one hidden layer and watch the boundary bend. Then sweep hidden size and regularization, one knob per run, and predict the boundary shape before each run finishes. Check the gradients against numerical tests on a tiny input before you trust the sweep.

What Carries

Use toy problems to see the mechanism and real data to see the failure modes. Two dimensions hand you the whole loop and remove one thing. A point on the spiral has two coordinates and no neighbors, so the model has no spatial structure to exploit. Pixels have neighbors, and the next architecture takes that structure for free rather than making the model learn it.