One Template Per Class Is the Whole Limit

A linear classifier throws away the training set and keeps a weight matrix instead. Nearest neighbor kept every image and changed only the comparison rule. This model reverses that trade, and it is the first real model in the course. The price of the compression is one averaged template per class.

The whole model is one matrix multiply

The function is blunt. Flatten the image into a column, multiply by a weight matrix, add a bias, read one score per class. For CIFAR-10 that is 3,072 pixel inputs and 10 class scores.

scores = W @ x + b
predicted_class = argmax(scores)

The model is simple enough that shape mistakes are embarrassing and common. Write the shapes down once and check the code against them.

x:      3072 x 1
W:      10 x 3072
b:      10 x 1
scores: 10 x 1

If you can write those four lines without looking, you understand the model.

The bias moves the decision boundaries. Without it, every boundary must pass through the origin. A common trick appends a constant 1 to every input vector. Then W absorbs it as one extra column.

That is bookkeeping. Clean bookkeeping matters, because backpropagation will depend on boring and correct shapes.

A row of W is a class template you can reshape into an image

Each row of W is one class template. It likes some pixels, dislikes others, and returns a single number. That number is a dot product between the row and the image.

The row has the same length as the flattened image. Reshape it back into image shape and you can look at what the model learned about that class. In early vision that window matters, because the model stays small enough to inspect all the way down.

Geometry comes before probability here. The classifier cuts pixel space with planes, and each score reports which side of a boundary holds the image. The size of the score reports the strength of that answer.

One template must average every visual mode of the class

One template per class is also the limit. Cars appear red and blue, front-facing and side-facing, and half-occluded. One row of W must average all of those modes, and the average is a blurry compromise.

Those modes are the pressure list from the overview note. Pose, scale, color, and occlusion are what a single averaged row cannot hold at once.

The weights also use whatever separates the training set, including background and scale. A feature that splits the training images without generalizing still lowers the loss. The failure stays legible, which is the useful part. When one boundary cannot separate the visual modes, you know why deeper representations become necessary.

SVM and softmax punish different stories

Multiclass SVM loss asks more than whether the correct class scores highest. It asks whether the correct class beats every rival by a margin.

The margin is the point, because a barely correct score is fragile. A score that wins by enough leaves room for a rival to move.

A rival sitting inside the margin costs loss. Once every rival clears it, that example contributes zero data loss and SVM stops pushing on it.

Softmax starts somewhere else. It turns the same scores into a probability distribution. Cross-entropy then charges the model when the true class gets low probability.

Softmax also hands you a number that looks like confidence. High probability and calibrated probability are separate claims, and the held-out split settles which one you have.

SVM keeps rivals away from the correct class. Softmax piles probability mass onto it. Both make bad scores expensive, and they press on the same ten numbers in different places.

The difference shows up in the training curves. SVM can ignore examples that are already safe. Softmax keeps shaping confidence long after the ranking is correct.

Accuracy counts whether the top label matched. Loss reports how strongly the model was wrong or right, and the optimization note reads that curve.

Regularization states which fitting solution you meant

Without regularization, many weight settings fit the training data, some sharp and brittle, others smaller and smoother. Regularization names which one you prefer.

L2 penalizes large weights, which stops one accidental pixel from carrying the decision. The data loss says fit the labels. The regularizer says fit them in this kind of way. That preference is the model you meant to build.

The Builder Test

Run three checks before you reach for a bigger model.

• Reshape each row of W into image shape and look at it.
• Inspect the class confusions and ask which two templates overlap.
• Compare train loss with validation loss on the same run.

Read the gap between those two numbers. A wide gap means the model fit noise instead of structure.

What Carries

Scores become behavior only through a loss. The loss defines what the model pays attention to, so choose it with care. It says how strongly the model is wrong. The next question is how the parameters move because of it.