Generative modeling is a set of contracts, and the contract you name chooses the model family. Name the family first and you spend the project defending a choice you cannot test.

The job is to learn enough structure from the training distribution. With that structure you can assign probability, sample, reconstruct, interpolate, edit, and reason about missing data. The phrase make new images covers all of that and distinguishes none of it.

Each job in generative modeling asks for a different contract

Likelihood estimation, sampling, reconstruction, representation learning, controllable editing, compression, and missing-data reasoning are separate jobs. Each one names a different output and hands you a different kind of control over the image distribution. Treat them as one job and every comparison afterward is muddy.

A job that needs likelihood takes an explicit probabilistic model, one that scores data cleanly. Sharp visual samples sit closer to adversarial and diffusion-style objectives. Latent-variable methods are the ones to look at when the job is a compact representation.

Each family answers a different question

The generative-models note walked the mechanics, PixelRNN and PixelCNN, autoencoders, VAEs, and GAN training. Underneath those mechanics sit three different objectives.

An autoregressive model predicts the next visible piece from the pieces before it. A VAE compresses an image into a latent variable, decodes it back, and keeps that latent space sampleable. The GAN objective trains a generator until a discriminator cannot separate its samples from real data. Each objective carries its own strengths and its own failure modes.

A latent space is a bet that hidden factors explain the pixels

Latent-variable models bet that lower-dimensional hidden factors explain high-dimensional images. An organized latent space makes interpolation and sampling behave, and a tangled one reconstructs but does not generalize gracefully.

Reconstruction error looks the same in both cases. Walk a path between two latents and watch whether the decoded images stay plausible. When representation is the job, that walk is the evaluation.

Sampling makes the model show its distribution

A classifier hides behind one accuracy number. A generative model must put its distribution on the page. The samples show blur, mode collapse, missing global structure, texture shortcuts, and memorization. You can see the failure before you compute anything.

Name the metric, then name what the metric hides

Likelihood, sample sharpness, diversity, speed, controllability, reconstruction, and downstream features each judge a different promise. Sharpness does not report diversity, which is why the list carries both. Every metric on that list hides a failure, and a second measurement has to catch it.

The classification note warned about the clean accuracy number, and a clean likelihood number buys no more trust. A shallow pass compares samples by taste. A serious pass compares promises by evidence.

The Builder Test

Before you name a family, write three lines. The output the system needs goes on the first. The metric that judges it goes on the second, and the failure that metric hides goes on the third. If the third line stays empty, the comparison is not ready to run.

A family chosen because its samples looked good in a talk fails the first line.

What Carries

The overview note opened on one question, which is what representation survives an image nobody arranged for the model. Samples are that representation made visible, and generation answers the question with pictures rather than a score. Name what your representation must survive: pose, lighting, scale, background, occlusion. Then choose the machine that survives it.

An autoregressive image model buys an exact likelihood. It pays twice: once in the order it commits to, once in the sampling speed it gives up. The mask is what keeps that contract honest.

PixelRNN models an image as a sequence. Predict the next pixel from the pixels before it, then repeat. A model that assigns high probability to real images learned something about the image distribution.

The generative models note named this family and stopped at the price. Two things set that price: a product of conditionals, and a mask that enforces the order.

Factor the image into conditionals and the objective stops being vague

An autoregressive model decomposes the probability of an image into a product of conditional probabilities. The model predicts each pixel from the pixels that came before it. Train it by maximizing likelihood, or by minimizing negative log likelihood. Those are the same objective written two ways.

The objective is explicit, which is cleaner than many generative methods manage. You know what you optimize. The number the model reports is the number the training loop moved.

The order is part of the model

Choose the order and you choose the prediction problem. Each pixel inherits one context, the pixels ranked before it, and nothing else. Change the ranking and every conditional answers a different question.

Every prediction carries a context and a probability. The visualizing ConvNets note left confidence as a claim about the model itself. Here the likelihood states the context it assumed and the quantity it computed. A general sense of image quality decides nothing.

Masked convolutions stop the model from seeing the pixel it must predict

PixelCNN uses masked convolutions so each prediction sees only the pixels the order allows. A convolution kernel covers a neighborhood in every direction. The mask zeroes the taps at the target position and after it. Causality then holds inside a convolutional architecture.

Drop the mask and the model reads the answer it must predict. Training loss falls for the wrong reason. The note on setup failures asked whether the train-time and test-time paths agree. A leaking mask breaks that agreement inside one forward pass, because at sampling time those pixels do not exist.

The mask also makes training parallel. A recurrent scan walks positions one after another. Masked convolutions score every position of a training image in one pass, and the autoregressive factorization survives.

Sampling pays the bill for exactness

The bill arrives at generation time. Every pixel depends on the pixels before it, so sampling runs one position at a time. An image takes as many forward passes as it has pixels.

Generate all pixels at once and the cost disappears along with the model. That breaks the contract. You trained each conditional with the earlier pixels in hand, and independent sampling removes them.

Global structure must travel through many conditional decisions

Local consistency comes free. Each prediction conditions on nearby pixels that are already fixed, so edges line up and texture stays continuous.

Global structure has to travel. A commitment made in one corner of the image reaches the far corner through a long chain of conditionals. Every link in that chain can drop it.

That is one reason later systems add stronger global representations: latent variables, attention, or diffusion dynamics. Each of those gives long-range structure a route that skips the pixel-by-pixel chain.

The Builder Test

Take one output position. Draw the receptive field around it and mark every input pixel the mask allows through. Then look at the target position and everything after it in your chosen order.

If any of those taps survive the mask, the model trains on the answer. Fix the mask before you read another loss curve.

What Carries

Diffusion and transformer-style token models run most image generation now. The lesson outlives the architecture. A generative model is accountable when its factorization, its objective, and its sampling procedure are all stated and all agree.

An explicit contract like this one is easy to judge. The harder job comes first: name the contract the system needs, then choose the family that can sign it.

The field moved from AlexNet to ResNets to ConvNeXt to ViTs, and the training habits did not become optional. Old ConvNet advice still works because the failures it prevents did not change. Bad data, wrong normalization, unstable learning rates, leakage, weak baselines, and uninspected errors still break strong models.

Most practical ConvNet work is preparation. You clean the old machinery well enough that the model gets a fair chance.

Augmentation states which changes must not change the label

An augmentation is an invariance claim written in code. Random crops, flips, color jitter, and scale changes each assert that some transformation leaves the label alone. The classification note named what a vision system must stay invariant to: pose, lighting, small translation, occlusion, and clutter.

The constraint is taste. Augmentation must not change the task. Flip a street-sign dataset blindly and you can change the answer. Turn medical scans casually and you teach the model a lie.

Normalization must be identical between train and test except for stochastic augmentation

Subtract means. Scale channels. Run the same pipeline at test time, with the random transforms switched off. A model trained under one input distribution and served under another must improvise. Nothing crashes when the two pipelines disagree, so the bug shows up only as a worse number.

Stacks of small filters beat one large filter

Stacks of small filters build large receptive fields with fewer parameters and more nonlinearities than one giant filter. That is why repeated 3 by 3 convolutions became a durable pattern. The motif matters when you pick a backbone, because it sets parameter count at a fixed receptive field.

The learning rate is a schedule that changes during training

Warmups, step decay, cosine schedules, and careful restarts express one instinct. The correct step size at the start is the wrong step size later. An unstable rate looks like a bad architecture from the outside. Watch the curve before you blame the model.

A dull recipe beats a clever mess

Start from a known backbone and match preprocessing to it. Use pretrained weights when a good one exists. Train a simple head and keep the shapes honest. Get a validation baseline that behaves sanely before you tune anything.

Stack tricks after that. Augmentation, schedules, weight decay, label smoothing, mixup, and stronger pretraining can all help. Each one is easier to reason about once the baseline holds still.

The one-change rule is the same discipline the training note defined, one level further along the pipeline. Change augmentation, optimizer, batch size, and architecture at once, and you can win the run while learning nothing. That trade is fine for a desperate leaderboard run. It is bad for understanding.

Log every run. Save the config beside its checkpoint, track data versions, inspect calibration, and test on corruptions or distribution slices. Those checks guard the same split the classification note sealed. A dataset that changes under a run makes every comparison meaningless.

If you cannot name the dominant error mode, changing the architecture is theater

Accuracy reports the scoreboard. Error analysis decides the next intervention. Look at false positives, false negatives, confusion pairs, rare classes, lighting failures, background shortcuts, and corrupted inputs.

One error mode usually dominates. A newer model does not fix it until the boring checks pass.

The Builder Test

Name the dominant error mode from your last run in one sentence, with the slice it lives in. Without that sentence, run error analysis before you touch the architecture. The sentence is the permission slip for the change.

What Carries

Dull work buys the right to test an interesting architecture. Discipline decides whether a result is real. The last two notes ask what result you were trying to produce.

The gradients that trained the model are the gradients that break it. A defense means nothing until you name the attacker.

A tiny perturbation, invisible or irrelevant to a human, moves a network into a confident wrong prediction. The overview opened this unit with a pressure list: pose, lighting, scale, background, occlusion, class imbalance. An adversary is that list with intent. It searches for the one shift you never tested.

Differentiability makes the attack cheap

A neural network is differentiable from the loss back to the pixels. Training walks that path to the weights. The attack walks the same path to the input. It nudges each pixel toward a higher target score, or a lower score on the correct class.

The visualization note computed that same input gradient and drew it as a saliency map. Saliency asks which pixels the model leans on. The attack asks which pixels to move. One backward pass answers both.

A high softmax score is confidence under the learned representation

A high softmax score reports confidence under the learned representation. It carries no claim about whether the image is unambiguous, safe, or aligned with human sight. A perturbation can push the model across a decision boundary while a human sees the same object.

That gap decides how much weight the score deserves. It matters most in systems that touch money, safety, identity, health, or physical control.

Clean held-out accuracy is the number the classification note warned about, and the adversary attaches an optimizer to it. Your held-out set arrives by sampling. The adversarial set arrives by search.

FGSM takes one step, PGD takes many

The Fast Gradient Sign Method takes one step in the direction that increases loss with respect to the input. PGD takes many smaller steps and projects each one back inside the perturbation budget. It is usually the stronger first-order attack.

One step assumes the loss surface stays straight across the whole budget. Many small steps drop that assumption and follow the curve, which is the whole difference between the two. A defense tested only against FGSM reports a number PGD does not support.

A defense without a named attacker is a wish

A robustness claim only means something inside a threat model. Name what the attacker can change: pixels within an L-infinity bound, a physical sticker, lighting, camera angle, or compression. Say what the attacker can examine, including your weights, your preprocessing, and the defense itself. The success condition decides the rest, because any wrong label and one specific wrong label are different attacks.

Change any one of those three and the defense changes with it. A method that holds under an L-infinity budget can fail completely against a sticker on the object. Write the threat model before you write the defense.

A defense that hides gradients looks strong until someone adapts

Gradient masking, image compression, and naive preprocessing all lower attack success on the first try. They damage the gradient the attacker needs. The attack then fails for a reason that has nothing to do with robustness. A weak evaluation reads that failure as a defense.

An adaptive attack that knows the defense reads it correctly. It usually shows the model was only hard to attack with the first method you tried. The honest standard is harsh: evaluate against attacks that know your defense.

The options that survive that standard are adversarial training, certified robustness in limited settings, better evaluation protocols, and system-level safeguards. Each one costs training time or narrows the setting where the claim holds.

The Builder Test

Before you ship a vision model, run four evaluations and report them in one place.

• Clean held-out accuracy.
• Accuracy under corruptions.
• Accuracy under distribution shift.
• Accuracy under an adaptive attack that knows your defense.

A defense that moves only the first number has moved nothing. After launch, keep watching real failures. The threat model you did not write is the one production finds.

What Carries

Accuracy says how often the model is right on the dataset you chose. Robustness asks how badly it fails when the world stops cooperating. The reliability work that catches most of your real failures is older and duller than any attack. Do that work first, then earn the right to worry about the attacker.

NeRF stores a scene as a function you query. Posed photographs are enough supervision, because rendering is differentiable and the gradient reaches the field through the image.

Give the network a 3D location and a viewing direction. It returns density and color. Render enough rays through that function and you have a new view of the scene. The pipeline stores no mesh and no voxel grid.

The scene becomes a field you query instead of a table you store

The field maps coordinates to radiance and density. Density says whether something is there. Color says what light leaves that point in a given direction.

The representation stays continuous, so you can query between sampled points. A grid stores answers at fixed places, and a coarse grid caps the detail you can get back. A function computes an answer wherever you ask.

Rendering supplies the training signal

A ray leaves the camera and passes through the scene. The renderer samples points along the ray and asks the field for density and color at each one. It accumulates those samples into one predicted pixel. Training compares that pixel to the captured pixel at the same place.

The backward chain here is the one from the backprop note, extended through image formation. Every accumulation step is a differentiable operation, so the gradient walks back through the samples and into the field weights. Backpropagation updates the field until rendered views match observed views.

The loss rule from the linear classifier note holds without change. Posed photographs are the labels. Wrong means a rendered pixel that disagrees with the captured one. No 3D ground-truth mesh enters the loop.

View dependence is a feature of the representation

Color depends on viewing direction, so one point can look different to two cameras. That is how the field carries highlights and other view-dependent effects. The scene lives in the parameters of a function, and you reach it by querying along a ray.

The overview note listed the pressures a representation must survive, and pose sits first on that list. One field serves every camera, so view consistency becomes a property of the representation. The test is a camera pose the model never saw. Render it and look for geometry that drifts.

Positional encoding lets an MLP hold high-frequency detail

A plain MLP struggles to hold high-frequency detail when you feed it raw coordinates. Positional encoding maps each coordinate into a set of sinusoidal features. Those features give the network a handle on fine structure. The change happens before the first layer.

The architecture note called a design a bet about what structure the data has. That encoding places the bet at the input. The bet is that the scene holds fine structure the raw coordinates make hard to reach.

Classic NeRF pays for exactness on every ray

Each pixel costs many samples along its ray, and each image costs many rays. Per-scene optimization repeats that bill for every new scene. That makes classic NeRF slow to train and slow to render.

It also needs good camera poses and enough coverage. A pose that is off puts the ray in the wrong place. The field then fits the scene to that wrong ray.

Later work improved speed, generalization, and editing. The original design still earns study, because geometry and appearance become one differentiable training problem.

The Builder Test

Trace one pixel from the camera to the loss. Pick one pixel in one training image and name the ray it comes from. Write down the samples along that ray and the accumulation that turns them into a color. The captured pixel at the other end of that comparison is the label.

Now move the camera pose by a small amount and walk the same chain. The ray starts somewhere else, every sample moves with it, and the loss charges the field for the difference. You pass this test when you can point at the exact step where pose error becomes geometry error.

What Carries

A scene can be learned as a function when rendering gives the model a way to be wrong. Before you trust any new representation, find the operation that makes its error measurable. A representation shift changes what a model can express, and it changes nothing about how easily gradients turn against it.

Each generative family buys one property and pays in a fixed direction. Name the metric first, and the family follows.

The linear-classifier note treated the loss as the real decision, because the loss decides what a model attends to. Generation applies the same discipline to the metric. Until you name one, sample quality stays a matter of taste.

Generation asks which images can come from this data at all

Classification asks which label fits this image, and a classifier answers with one accuracy number. Generation has no such number to stand behind.

A generative model has to put samples on the page. The batch shows what the model treats as typical. It also shows which modes it misses, which details it blurs, and which artifacts it repeats.

Autoregressive models buy clean likelihood and pay in sampling speed

PixelRNN and PixelCNN model an image as a sequence of conditional predictions. The model predicts each pixel from the pixels before it, in one fixed order. That order gives a likelihood you can compute directly.

Sampling has to follow that order, one pixel at a time. The cost lands on generation, and it lands there again on every sample you draw.

Autoencoders compress, and VAEs make the latent space sampleable

An autoencoder maps an input to a latent code and rebuilds the input from that code. The code keeps the information reconstruction needs and drops the rest. Sampling from that space is a separate job, and a plain autoencoder does not organize it for you.

A variational autoencoder puts a distribution over the latents. The encoder predicts that distribution, and the decoder rebuilds from samples of it.

The loss is a negotiated settlement between two terms. The reconstruction term wants faithful outputs. The KL term wants the latent distribution to stay close to a simple prior.

Press the prior too hard and reconstructions suffer. Press reconstruction too hard and the space stops being worth sampling. Most VAEs settle where the samples blur.

The check is cheap. Decode two nearby latent points and look at both images. If the pair stays sensible, the space is organized enough to sample from.

GANs sharpen samples and destabilize training

A GAN runs a generator against a discriminator. The generator makes images, the discriminator separates generated images from real ones, and the generator improves by fooling it.

Sharpness comes out of that game. The discriminator rejects unrealistic samples, so the generator learns from a signal closer to perception.

The same game breaks the training. The generator can exploit the discriminator instead of covering the data, or collapse onto a few modes. The signal keeps moving while the model learns, which is why GAN evaluation stays hard.

Likelihood and sample quality are different scores

One model assigns good likelihood and still produces blurry samples. Another produces sharp samples and makes likelihood hard to compute. Rank the same models by one score and then the other, and the orders disagree.

Comparing autoregressive models, VAEs, GANs, and diffusion-style methods needs a named metric. The candidates are likelihood, perceptual quality, diversity, controllability, and speed.

Each metric stays quiet about its own failure. Likelihood says nothing about blur, and perceptual quality says nothing about missing modes or memorization.

Generation turns into copying when the data, the objective, or the evaluation lets memorization pass as sampling. The classification note named leakage and near-duplicate checks for the same reason. Both ask what the model kept.

A sharp sample is the model being confident about the data distribution. The visualization note put that warning on a class score, and it carries over here. Confidence is confidence under the representation the model learned.

The Builder Test

Take a model you trained and generate a batch. Judge the batch on the one metric you named before you looked at it. Then write down the failure that metric cannot see.

The likely failures are already known. Autoregressive sampling runs slow and VAE reconstructions blur. GAN training drops modes, and diffusion needs a run of denoising steps for every image it makes.

The batch then needs a second check, chosen against the failure you wrote down. Count distinct modes when the metric rewards sharpness. Compare each sample against its nearest training image when the metric rewards fidelity.

If the batch fails that second check, the score it earned does not mean anything yet.

What Carries

Samples are the honest surface of a generative model. One good sample is the cheapest thing that surface can give, and a collapsed model still produces one. Evidence starts at a batch, judged on the metric you named before generating.

Every family here learns a distribution over images. The next representation learns one scene, with rendering supplying the label.

A vision transformer trades convolution's built-in locality for learned routing, and the training recipe pays the difference.

The attention note left routing as one operation inside a recurrent model. A transformer promotes routing to the backbone. Recurrence goes away, and the model reads every token in parallel. In language the tokens are words or subwords. In vision they are image patches.

Self-attention makes the path between two positions short

Inside self-attention every token reads from other tokens. A patch gathers evidence from a distant patch in one step. A word gathers evidence from earlier or later words, depending on the setup.

A recurrent model carries one compressed summary forward, and that compression is the ceiling. Attention removes the carry.

Stack the blocks and the effect compounds. Each layer builds a table of relationships between the visible positions. Then it updates every patch with the evidence attention routes to it. Local patch evidence becomes global object evidence.

Patch size is a modeling decision you make before training

Patchification is the entry point. The model cuts the image into patches, embeds each patch, adds positional information, and feeds the sequence into transformer blocks. After that step the architecture looks language-like.

The patch is a rough unit. Small patches keep fine structure and lengthen the sequence, and large patches cut compute and lose detail. Tokenization is part of the model, so set the patch size against the smallest structure your label depends on.

The patch grid also changes how the pressure list from the overview note reaches the model. Translation now moves evidence between patches instead of sliding it under one shared filter. Scale decides how many patches an object covers.

Attention needs position because it does not have one

Attention is mostly permutation-friendly. Shuffle the tokens and the operation barely notices. The model needs positional information to know where a token came from.

In vision that information carries weight. A patch at the top of the image and the same patch at the bottom are not the same evidence. Position is the only thing that separates them.

Multi-head attention gives the model several routing channels

Different heads learn different routing patterns. One head stays local while another connects object parts, and a third reaches for global context. The gain is capacity for several relation types at once.

Do not over-romanticize a single head. A tidy caption for one head is something a person writes after training.

Less built-in bias means the data must teach what the architecture no longer assumes

The convolution note named the two assumptions a ConvNet ships with: locality and weight sharing. Nearby pixels belong together, and the model reuses one filter across every position. Patch tokens keep much less of both.

A vanilla transformer must learn that structure from data and pretraining. Vision transformers became convincing alongside scale, stronger augmentation, large datasets, and self-supervised pretraining. Architecture details hand back some of the prior too, and the training recipe carries a lot of the win.

The same recipe teaches whatever else the data contains. The visualization note's shortcut hunt transfers to attention maps without a change of tools. A route can land on the background, a repeated texture, or a dataset artifact and still produce the right label. A clean attention map is confidence under the learned representation, and nothing more.

Small data, tight latency, and dominant local texture still favor ConvNets

I still reach for a convolutional backbone when the data is small and the latency budget is tight. Local texture and spatial hierarchy can dominate a task, and convolution already assumes both.

A transformer is a trade. Take it when distant parts of the image decide the label and you can pay the data bill.

The Builder Test

Name the smallest structure your label depends on. Measure it against your patch size. If that structure fits inside one patch, the grid is fine. If it straddles several patches, attention must reassemble it, and that reassembly is what you test first.

Then name what supplies the locality prior you removed. If you cannot name one, you dropped a prior and put nothing in its place.

Inspect what a patch attends to. If the route does not match the evidence a person uses, the model solves a different task.

What Carries

Carry the routing view. A transformer learns which pieces of the image must talk to each other. The architecture says almost nothing about which pieces those are.

Price the prior before you remove it. When nothing in the recipe teaches locality, the patch grid buys flexibility you cannot use.

Routing and recurrence both end at a label. The next family must produce the image itself.

Attention replaces one compressed state with a different context vector for every step. The mechanism is retrieval.

A fixed image vector asks one representation to serve every word

A captioning decoder that reads one fixed image vector asks that vector to serve every word it generates. The word dog and the word ball need different regions of the same image. One summary cannot hold both at full strength.

The recurrent networks note priced that compression. One hidden state gets rewritten at every step, and everything the last word needs has to survive the trip. Attention lets the decoder build its context after it knows what this step is asking for.

The change has a price. The decoder now keeps every region available and scores all of them at every step.

Queries ask, keys advertise, values answer

A query states what this step needs. Keys describe what each stored item holds, and values carry the information that comes back.

Scores compare the query to the keys. The score between a query and a key decides how much of that value enters the context. Each step weights the patches it needs and gives the rest little weight.

That makes attention a differentiable retrieval and routing mechanism. The operation is learned and soft, and it usually runs in parallel.

Vision-language tasks buy grounding from it

In image captioning, the decoder looks at the regions that matter as it produces each word. When it generates red car, it must look at the region holding the car, not the sky.

That is grounding, and visual question answering needs it too. The language side gets a mechanism that connects each text decision to visual evidence. The relations it routes over can be spatial, semantic, or cross-modal.

The model is also more flexible than one fixed image vector, and it is easier to inspect. You can read which region fed which word.

Attention weights are evidence about routing and nothing more

Attention weights are useful evidence about routing. A map shows which region the model scored highly for a word. It does not show that the region caused the word.

The visualization note set the rule for probes. A probe earns its place when it produces the next test. An attention map lives under that rule, so read one map, then write down the question it raises.

Attention moved the bottleneck from memory to routing

Self-attention lets tokens attend to other tokens in the same sequence. Transformers stack that operation with feed-forward layers, normalization, residual connections, and positional information.

The old bottleneck was memory compression. The new bottleneck is routing, compute, context length, and data. Those are the limits self-attention inherits when it becomes the backbone.

Architecture is a bet about what structure the data has. Attention bets that the structure worth having is which piece matters at this step.

The Builder Test

Take one generated word from your captioning model. Print the top attention weights and the regions they select.

Then trace the whole path: query, keys, values, weights, and the evidence that came back. Ask whether that evidence supports the word the model produced or a different word nearby. If a nearby word fits the selected region better, you have found the next test to run.

What Carries

Routing decides what the model looks at. Looking is still not understanding.

Routing is the operation that pays here. The next move is to make routing the whole backbone.

An RNN carries one compressed summary forward, and that compression is both the mechanism and the ceiling. The hidden state is the only thing that reaches the last step. Whatever the model knows about the past must fit inside it.

Images are spatial. Captions are sequential. A vision system that describes or acts over time needs memory. Recurrence is the old answer: read one token at a time and carry a state forward.

The hidden state is a notebook rewritten at every step

At each step the model reads the current input and the previous hidden state, then writes a new one. That is the whole recurrence. The notebook is rewritten at every step and never extended.

A character-level model shows the shape at its smallest. The input is one character. The output is a distribution over the next character. Every earlier character reaches that distribution only through the state.

That summary is not a transcript. A long-range dependency that fails is information the representation did not keep.

The same transition weights recur at every step

One transition function runs at every time step. The same weights read the first element of the sequence and the last.

Convolution reuses one filter across space. Recurrence reuses one transition across time. Both are bets about structure, and this bet is that the past compresses into a single state.

Sharing pays twice. It handles sequences of variable length, and it learns temporal patterns that repeat at any position. It also builds one long chain. A gradient that reaches the first step travels back through every step in between.

Backpropagation through time makes the chain explicit

To train the transition, unroll the network across time and run backprop on the unrolled graph. The same parameters appear at every step, so their gradients accumulate across the whole sequence.

That backward path is the chain rule the backprop note made concrete, now multiplied along time. Gradients vanish or explode as they pass through many repeated transitions.

The failure has one signature. The model keeps recent tokens and forgets older ones that still matter.

Gates buy a stable path for information across time

An LSTM splits the state in two. A memory cell holds what the model keeps. The rest of the network sees only the exposed hidden state.

Gates control the write, forget, and read operations, and the model learns each gate. It decides what enters the cell and what decays inside it.

That is controlled memory, and it buys a more stable path for information and gradients across time. The model keeps important information longer and lets irrelevant noise decay.

Transformers displaced RNNs for many tasks, and the state lesson outlasted them.

Image captioning is the bridge from one label to a sequence

Image captioning starts with visual features from a CNN. A sequence model then produces words one step at a time.

The two parts have clean jobs. The encoder extracts visual evidence. The decoder turns that evidence and its own previous words into the next word.

The image features initialize or condition the decoder. The visual evidence enters once and must last to the final word.

That split is the frozen backbone with a new head from the transfer note, moved from space to time. Both reuse one visual representation for a new output.

The Builder Test

Write down what the hidden state must still hold at the last step. Name a specific item.

Then count the steps between where that information enters and where the model uses it. That count is the length of the trip the state must survive.

Build the smallest version of that trip. Make a sequence whose answer depends on one early step, then read the output at the last step.

A model that holds on short sequences and fails on long ones has reported the answer. The state dropped the early item.

Check the objective too. A loss scored one step at a time can stay low while the whole caption is wrong.

A weak review says the model has memory. The strong review names which memory is too compressed to survive.

What Carries

Whenever a model reads a sequence, name what it carries forward and what it drops before the decision. The dropped part is where the failure starts.

Gates lengthen the trip a state can survive. The bottleneck is still one state, and the next note removes it instead of enlarging it.

Two numbers decide the transfer plan: the size of your labeled set, and its distance from the pretraining data. Everything else follows from those two, including where you freeze. The common failure is reaching for a pretrained backbone by reflex, without asking whether the borrowed features match the job.

Training a large vision model from scratch is usually the wrong default

Random weights hold no visual features, so scratch training pays for edges and textures you can borrow. Start from a model that already learned those features on a large dataset. You then have two moves: use it as a feature extractor, or fine-tune part of it for your task.

Both moves rest on the same bet, which is that the source task learned features your task can reuse. Early visual filters, textures, shapes, and object parts transfer even when the final labels change. You replace the classifier head because it was trained to name a different set of classes.

Data size and domain similarity give you four cases

Similarity here means how close your images are to what the pretrained model already saw. Small and similar is the easy corner. A frozen backbone and a new head are enough. A large similar dataset earns more room, so fine-tune deeper and trust the result more.

The dangerous corner is small and different. The borrowed features can be wrong for your images, and you do not have enough data to learn better ones. Large and different makes full fine-tuning plausible, and pretraining on your own domain becomes a real option.

Feature extraction is fast and has a ceiling

The pretrained network turns your images into embeddings, and a small classifier trains on top. This is fast and stable, and it overfits a small dataset less easily than a full fine-tune. The ceiling arrives when the frozen features miss something specific to your domain. The head cannot invent what the backbone never encoded.

A frozen backbone bets on coverage. The pressures the source images spanned, pose, lighting, scale, background, occlusion, must also cover yours. When they do not, the head reads a representation that already discarded the signal you need.

Fine-tuning must adapt the features it inherited

Fine-tuning updates part or all of the pretrained network. The features you want are already in there, so every update is a disturbance you must keep small. Use a smaller learning rate for the pretrained layers and a larger one for the new head. A reckless learning rate erases the head start you paid for.

Unfreeze from the top down. Early layers hold generic edges and textures. Later layers hold dataset-specific semantics, which is where your task differs most. That order is a default, and error analysis overrides it.

Fast improvement is exactly when leakage hides

Transfer makes performance rise fast, and a fast rise covers a dirty split. Check that near-duplicate images do not cross train and validation before you trust the validation curve. The split is still the only instrument you have, and a borrowed backbone does not repair a broken one.

Check your preprocessing against what the pretrained model expects. That match is the same train-test path agreement the note on setup failures demanded. You now inherit that contract from a model you did not train.

One more leak sits upstream. The pretraining set can already contain your evaluation images, and no split of your own data will show it.

The Builder Test

Freeze the backbone and train the head. Record the errors on the hard slices. Pull the confident mistakes first, which is the same inspection the visualization note used to catch a shortcut. Unfreeze deeper blocks only when that analysis shows the representation itself is the limit.

What Carries

Transfer is a bet that the source features are reusable here, and the error slices are how you settle it. Transfer moves a representation between tasks. The next problem needs a representation that moves through time.

A visualization earns its place only when it changes a decision. Collect different data, fix a shortcut, reject a deployment, add a test. A picture that moves none of them is decoration.

The useful job is narrow. A probe catches the model using the wrong evidence while you can still act on it.

Feature-space neighbors show what the model thinks is similar

Pull the nearest neighbors of an image in raw pixel space and you retrieve similar color and similar background. Pull them in the feature space of a trained ConvNet and the neighbors share meaning instead. That difference is the finding. The network built a space where visual meaning is easier to compare.

Run the same retrieval across the training set and it pays a second time. Tight clusters that mix two labels mark unclear class boundaries. Exact repeats mark duplicates. An image sitting deep inside the wrong cluster is a candidate for a bad label. The case worth an hour is the neighbor that looks wrong to you and correct to the model.

A t-SNE map generates questions and proves nothing

t-SNE compresses high-dimensional features into a two-dimensional map, and similar images often cluster there. Use it for intuition and for error inspection. The clustering is not evidence of structure, because compressing to two dimensions discards what it cannot fit. A cluster that surprises you buys one thing, which is a question you can settle with examples.

The valuable visualization is the one that catches the shortcut

A bird classifier can learn sky. A medical model can learn scanner artifacts, and a product classifier can learn background color. All three stay accurate on the split you measured. They keep that accuracy until the deployment distribution changes, and users meet the failure before you do.

The classification note sorted misclassified images by confidence before trusting any number. The shortcut hunt uses the same sort. Pull the confident mistakes first, because a confident mistake is where a shortcut hides.

Saliency is a probe, and every probe distorts

Saliency asks which pixels most affect the class score. Deconvolution-style methods and gradient visualizations try to show what a neuron or a layer responds to. Run either one over a ConvNet and you read the local shared filters the convolution note built.

The builder question stays concrete. The model uses the object, or it uses a shortcut: background, watermark, color cast, or border artifact.

Deconvnets, gradients, feature inversions, and t-SNE each distort the truth in their own way. None of them replaces a quantitative test or error analysis. The requirement is the same for all four. A probe earns its run only when it hands you a testable next question. That question points at a shortcut, a class confusion, or a feature dependence you can verify.

Confidence is confidence under the learned representation

A ConvNet can be confidently wrong on an image that looks meaningless to a human. The same model can flip on a perturbation a human barely notices. The representation it learned is not human perception.

A ConvNet stays useful under that limit. A confidence score is a statement about the model, and it holds only inside the representation the model learned. State the limit before anyone acts on the number.

The Builder Test

Take the class your model confuses most often. Pull feature-space neighbors for its confident mistakes and look at what those neighbors share. Name that shortcut in one sentence. Then write the quantitative test that falsifies it, and run the test before you touch the model.

What Carries

A probe leaves behind two things, a picture and a test. Keep the test and let the screenshot go. Once you can see what a representation learned, the practical question is whether you can borrow it.

Convolution writes one claim about images into the architecture: useful patterns are local, and they repeat across space. That claim is the prior. The note on hidden layers called architecture a bet about the structure of the data. Convolution is that bet written into the wiring.

A ConvNet refuses to flatten the image for as long as it can

A fully connected network treats an image like a long spreadsheet row. Width, height, and channels collapse at the door. Flattening keeps every pixel value and loses the arrangement.

A ConvNet holds on to the grid until the classifier. Nearby pixels are related. The same edge detector is useful in many locations. Convolution puts both facts into the architecture rather than making the weights find them again.

Classic stacks repeat conv, nonlinearity, and pooling, then hand the result to a classifier. Later stacks changed the filters, the depth, and the connections. The constraint stayed the same. A network must keep spatial structure while it turns pixels into class evidence.

A local receptive field is the whole first assumption

A convolutional neuron looks at one small region of the input. That region is its receptive field. A 3 by 3 filter asks a small question. A 5 by 5 filter asks a larger one and pays more parameters for the answer.

Early visual evidence is local. Edges, corners, colors, and textures do not need every pixel at once. Full connectivity in the first layer buys connections the image cannot justify.

Stack the layers and the region one unit sees grows with depth. The receptive field at the last conv layer is something you compute.

Weight sharing is where the efficiency comes from

The same filter slides across the image, so one learned detector runs at every position. A horizontal edge that matters in the top-left probably matters in the bottom-right, and one set of weights covers both.

The note on linear classifiers priced the other arrangement. One row of the weight matrix is one class template. That template has to average every pose, background, and lighting the class shows up in. Weight sharing is the opposite trade. One detector gets reused across many positions, so position stops eating capacity.

Without sharing, a first conv layer explodes into millions of parameters. With sharing, you learn one filter and apply it everywhere.

The output size formula decides whether the layers fit

Stride sets how far the filter moves each step, and a larger stride shrinks the output. Padding adds border values so filters fit at the edges and the spatial size survives the layer.

With input width W, filter size F, padding P, and stride S, one line gives the output width.

output_size = (W - F + 2 * P) / S + 1

A non-integer result means the layer does not fit cleanly. The same arithmetic decides memory and how fast spatial information shrinks. Track it per layer or architecture design becomes guesswork.

The backpropagation note called the boring bugs shapes, caches, and missing averages. This is the same discipline moved up from one tensor to a whole stack. A layer that does not fit is a shape bug you can catch before any data moves.

Pooling discards detail on purpose, so name the trade

Max pooling keeps the strongest activation in a region and drops the rest. It cuts compute and buys tolerance to small shifts.

The image classification note listed what a vision system must survive: pose, lighting, small translation, occlusion, and clutter. Convolution and pooling buy the small translation. Everything else on that list, plus class imbalance, still lands on the data and the loss. Locality says nothing about a cat photographed from above.

im2col is the systems compromise that made convolution fast

A convolution is many dot products between filters and local image patches. im2col unfolds those patches into columns, reshapes the filters into rows, and turns the layer into one matrix multiply.

The cost is duplicated input values, because the same pixel lands in several columns. The return is that the implementation runs on optimized matrix multiply kernels. Budget the duplicate copies next to the activations, since both fall out of the same layer arithmetic. This compromise, more memory for a faster kernel, appears everywhere in deep learning.

The Builder Test

Take the stack you plan to train and do the arithmetic by hand before you run it.

• Compute output width and height at every layer, and stop at the first non-integer result.
• Say what one unit in the last conv layer sees in the original image.
• Name what pooling discarded on the way there, and name the task that needs it back.

A blank answer on the third line means the pooling layer arrived by habit. Change it or defend it before you spend a training run on it.

What Carries

The prior is the advantage. A ConvNet starts with locality and reuse already assumed, and it spends its capacity on which filters are worth keeping.

A prior that does not match the data is a cost you already paid. Locality and reuse hold on natural images. On data whose useful pattern is global, or tied to one fixed position, the constraint returns nothing.

A matching prior still lets a network learn the wrong evidence. The next job is to look at what it learned, one image at a time.

A spiral in two dimensions is the cheapest place to watch a linear model fail and a hidden layer fix it. The data has two coordinates, so the decision boundary is a line you can draw and look at. The failure arrives as a shape instead of a cause you inferred.

The spiral defeats a straight boundary no matter how you tune it

The linear classifier maps each point to class scores and trains with softmax loss plus regularization. It draws straight boundaries and nothing else. The spiral classes wrap around one another, so no straight boundary separates them.

That failure is the reason to run the linear model first. A straight boundary losing to a curve names its own cause, so you stop looking at the learning rate and the data loader. The linear-classifier note priced this limit as one averaged template per class. In two dimensions the same limit is a straight line, and the spiral is built to beat it.

One hidden layer carves the space into regions

A hidden layer transforms the input before classification. ReLU units carve the space into regions, and the final linear classifier works on that learned feature space rather than the raw two coordinates. The visible result is a curved, piecewise boundary.

The boundary bends because the network learned a feature transformation. The model finds an intermediate space where the final decision is easier. At this size you watch that happen rather than accept it on report.

The toy contains the entire loop

Four ideas from earlier notes run at once in this toy. The score function turns points into class scores and the softmax loss says how strongly the model is wrong. Backpropagation delivers the blame to every parameter, and gradient descent spends it on an update.

• Initialize the weights with small random values so units learn different features.
• Run a forward pass to compute class scores.
• Compute the softmax loss and add regularization.
• Backpropagate the gradients through the score function.
• Update the weights with gradient descent.
• Monitor training and validation behavior, and trust no single number.

Once that loop is real in two dimensions, the larger image models stop being mysterious. They run bigger, more structured versions of the same loop.

Regularization moves the boundary where you can see it move

Turn the penalty down and the hidden-layer model overfits the toy data. A larger hidden size makes the boundary more flexible, and too little regularization makes it jagged around individual points. Capacity and penalty stop being quantities you infer and become a shape on the screen.

The training note makes the tiny-batch overfit test the first gate before scale. This is the same gate with the answer drawn. A hidden layer that cannot overfit a spiral with the penalty turned down has a broken wire. You find it before a real dataset is involved.

A small model you cannot explain is borrowed confidence

Every part is visible at this size: data, scores, loss, gradients, updates, and boundary. The spiral is unrealistic on purpose, because removing scale is what lets you inspect the mechanism.

Large models give confusion room to hide behind parameters. A confusion that survives the spiral only becomes more expensive on the next run. Explain the small case before you buy the large one.

The Builder Test

Run the linear model on the spiral first and confirm the limit yourself. Add one hidden layer and watch the boundary bend. Then sweep hidden size and regularization, one knob per run, and predict the boundary shape before each run finishes. Check the gradients against numerical tests on a tiny input before you trust the sweep.

What Carries

Use toy problems to see the mechanism and real data to see the failure modes. Two dimensions hand you the whole loop and remove one thing. A point on the spiral has two coordinates and no neighbors, so the model has no spatial structure to exploit. Pixels have neighbors, and the next architecture takes that structure for free rather than making the model learn it.

Training is an inspection loop. A network that cannot memorize ten examples has not earned a larger dataset.

The loop is cheap and the run is not. You examine gradients, overfit a tiny batch, and compare train behavior with validation behavior. Each reading decides whether the evidence supports the next move.

Overfit ten examples before you touch a real dataset

Take ten labeled examples and train until the model reproduces every one of them. A working pipeline memorizes ten examples without effort. If it cannot, the pipeline is broken.

The usual causes are wrong labels, gradients that never arrive, and regularization set too strong. A loss wired to the wrong tensor does the same, and so does broken data loading. Each survives a full run, which is the expensive way to find it.

The tiny-batch test is the gradient check from the backprop note, moved one level up the stack. That check tested one gate at a time. This one tests the whole pipeline against a task that is too easy to fail. Two more checks cost minutes: examine the augmentation, and make sure the metric means what you think it means.

Momentum and adaptive methods change how the update moves

Plain SGD zigzags through ravines. Momentum keeps a velocity vector that smooths the path and carries the update through consistent downhill directions. Nesterov momentum peeks ahead before it computes the gradient. RMSProp and Adam scale updates by recent gradient statistics.

Every one of these changes the shape of the curve the optimization note taught you to read. The data underneath stays exactly as wrong as it was, so validation still decides.

An optimizer cannot fix bad labels, broken preprocessing, or leakage

Adam, RMSProp, momentum, and learning-rate schedules make training smoother. They leave bad labels, broken preprocessing, impossible tasks, and leakage where they were. The setup note made preprocessing part of the model contract, and no optimizer renegotiates that contract.

When one optimizer is the only thing that makes a configuration work, inspect the configuration instead of tuning it. Treat the rescue as a symptom and go find what it covered.

Hyperparameter search needs a hypothesis, a budget, and a record

Search learning rate and regularization on log scales, because useful values differ by orders of magnitude. Run coarse sweeps first and keep them short. Refine only inside a region that already looks good.

The validation set that ranks those sweeps is the same split the classification note sealed. Every sweep spends a little of it, so keep the count. Record the settings you tried, what stayed fixed, and which images failed.

A sweep that runs until a number improves has produced a number and nothing else. Write the hypothesis before the run starts, and the result answers a question you can name.

Ensembles buy accuracy with complexity you must pay to serve

Different models make different mistakes, and that disagreement is where the accuracy comes from. You pay for it in training time, serving cost, and how hard the system is to explain. Take the trade when the gain is worth the cost, and name the cost first.

Change one knob at a time or the run teaches nothing

Two changes in one run produce one number and no way to attribute it. The run after that inherits the confusion and costs more to interpret.

A run with a broken validation split does not deserve a leaderboard. Scaling a confused experiment makes the confusion expensive, because longer runs hide wiring mistakes behind noise you paid for.

A new architecture does not rescue a broken training loop. Run the boring checks first, and the architecture becomes testable.

The Builder Test

Babysit the first run through four readings, in order.

• The first loss value, which rules out a loss wired to the wrong tensor.
• The tiny-batch overfit, which rules out the rest of the wiring.
• The gradient scale, which rules out dead gradients and an unstable initialization.
• The train-validation gap, which separates a model that learned structure from one that memorized.

A failed reading ends the run there. Fix that one thing, restart the four, and carry exactly one knob change into the next full run.

What Carries

The small run is the cheapest teacher you have. A big run teaches the same lesson slowly, and you pay for the delay in compute.

So shrink the next experiment until every part of it is visible. The smallest honest experiment has two dimensions, where nothing hides behind size.

Most training failures are setup failures. Bad input scale, bad initialization, missing normalization, careless regularization, and a mismatched loss make a good architecture look broken. A mismatch among those five wastes days that look like architecture problems.

Preprocessing removes friction that has nothing to do with the model

Mean subtraction centers the data. Scaling keeps feature ranges comparable. For images that usually means subtracting the training-set mean image, or the per-channel means.

Optimization works better when features sit on comparable scales and activations stop getting pushed into bad regions. Preprocessing removes variation the label never depended on, and it belongs before every other decision in the setup.

Initialization gives learning a fair start and nothing more

Weights too small and the signal vanishes. Weights too large and the activations or the gradients explode. Symmetric initialization makes every neuron in a layer learn the same thing. Random initialization breaks that symmetry, and it works only when the scale matches the layer.

Read the activation scale after the first forward pass. A collapse toward zero or a blowup names the initialization scale, before you touch the architecture.

Batch normalization and dropout change the function you evaluate

Batch normalization normalizes intermediate activations and learns a scale and a shift. It makes the network less sensitive to initialization and it can allow higher learning rates. It also changes the training system.

The layer uses batch statistics during training and stored running statistics during evaluation. You must track those running statistics and know what the layer does in each mode. Do not paste it everywhere.

Dropout disables units at random during training. The network cannot lean on one exact path, so it learns distributed representations that tend to generalize better. At test time dropout uses the full network with the correct scaling convention.

Forget the train() and eval() switch and you change the function you measure. The curve still moves and the number still prints, and both describe a model you are not shipping. I check the mode switch before I read any validation number.

Train-time and test-time paths must agree except where you meant them to differ

Preprocessing is part of the model contract. Subtract the training mean during training and you must subtract the same mean at test time. When the two input pipelines diverge, you evaluate the model in a different world than the one it learned.

The classification note put the trust boundary at the split. Leakage sends training information across that line. A mismatched pipeline sends the model across it. Both break the one instrument that reports the truth.

The only deliberate mismatch is training-only randomness such as augmentation or dropout.

Regularization is a budget on how much you trust the data

Every training setup decides how much to trust the data, the model, and the objective. Regularization is the explicit budget on that trust. It says how much freedom the model gets to fit the training data.

• L2 penalizes large weights, so the decision stops resting on a few fragile ones.
• Dropout removes units during training, so no single narrow path through the network carries the answer.
• Augmentation broadens the examples, so accidental correlations stop holding across the training set.
• Early stopping cuts the run before memorization dominates.

Each one answers the same question. How do we stop the model before it turns accidents into rules?

Training loss going down is a fact about the training set. The train-validation gap is the number to read. When training keeps improving and validation gets worse, the model is memorizing details that do not travel. When validation improves as training becomes a little harder, regularization is doing its job.

The loss is the game the model plays

The loss must match the task. Classification usually uses softmax cross-entropy or an SVM-style loss. Multilabel tasks need different treatment, and regression needs different objectives again.

Softmax cross-entropy and the SVM loss are the two the linear-classifier note weighed. Pick one and keep one name for it through the unit. A model tuned against a loss that states the wrong task moves numbers without moving the result.

The loss defines what counts as wrong, and every other control is tuned against that definition. Together they decide whether training learns structure or exploits convenience.

The Builder Test

Take a run you already have and plot the train and validation curves on the same axes. Then change exactly one thing: the preprocessing, the initialization scale, the regularizer, or the loss. Run it again and compare the gap.

An intervention that shrinks the gap while validation keeps improving is the one to keep. If the gap shrinks because both curves flattened, the change destroyed useful signal. Change one thing per run. Two changes and you cannot say which one moved the gap.

What Carries

Carry the gap. Generalization lives in the space between the examples you fit and the pattern you learn.

With the setup honest, the remaining question is how to run the training loop without lying to yourself. Keep the two curves on the screen while you answer it.

A hidden layer is a learned change of coordinates. Its only defense is that it makes the final comparison easier.

The linear classifier note ended on one averaged template per class. That template holds every pose and every background of a class in one picture, so it averages them. A hidden layer removes that specific limit.

It rewrites the input into coordinates where a straight boundary can do the work. The overview asked which representation survives an image nobody arranged for the model. This is the first stage where the model chooses that representation instead of receiving it.

Remove the nonlinearity and the stack collapses into one linear map

A neuron takes its inputs, computes a weighted sum, adds a bias, and sends the result through a nonlinearity. Take the nonlinearity out and two stacked layers multiply into one matrix. The depth stays on the page, and the model it defines is the template you already had.

That nonlinearity is the activation, and choosing it is a structural decision. ReLU, sigmoid, tanh, and their variants set how information and gradients move through the stack.

Depth is worth paying for when later layers reuse earlier features

Early layers pick up simple patterns. Later layers combine them into edges, textures, parts, and structure that belongs to a class. Each stage buys its inputs from the stage below, and only the first one reads pixels.

One averaged template cannot do this. It smears every mode of a class into one average. A stack keeps the modes apart, because a part detector can fire on two images that do not look alike.

Depth pays out only when the optimizer can train it and the data can support it. Otherwise the extra layers become capacity for memorization, instability, or shortcuts.

Activation choice is a decision about gradient flow

Sigmoid and tanh saturate. Once a unit sits in the flat region, the gradient it passes down is small. The layers under it learn slowly. ReLU is simple and often trains better, and it fails in its own way.

A unit that parks on the negative side passes no gradient, and it stays parked. That unit is finished learning for the rest of the run.

Universal approximation does not promise that SGD finds the function

Depth, width, and activation each set capacity, optimization difficulty, memory, speed, and overfitting risk. Universal approximation is a statement about capacity. It says a network can represent a wide family of functions.

It says nothing about whether SGD finds that function, whether the result generalizes, or what the training costs. Representable and trainable are separate properties, and generalizable is a third.

The useful question is whether this is the right amount of power for this evidence.

The Builder Test

Ask what changed between input space and hidden space. Take the examples the model gets wrong and find where they sit in the hidden representation. If they are still mixed in with the examples it gets right, the layer has not started doing work.

Then check which failures still look linear. Those are the cases the new coordinates left where they were.

The common failure is counting layers without asking what each one makes easier. When the data, the loss, or the optimization is wrong, more layers make the bug harder to see.

What Carries

Architecture is a bet about what structure the data has. A hidden layer is the smallest version of that bet. You claim that useful intermediate features exist in this data, and the training run settles it.

The bet raises the price of every setup choice. When the representation itself is learned, a bad setup looks exactly like a bad model. Audit the setup before you add another layer.

Backpropagation is the chain rule kept as bookkeeping. The forward pass computes values. The backward pass computes blame. Almost every bug in it is a wrong shape, a stale cache, or a missing batch average.

The calculus is one line. The work is keeping shapes, cached values, and gradient routes correct across many gates. Most implementation bugs are bookkeeping errors that still cost you a training run.

Every gate keeps a two-line contract

A network is a graph of small operations: add, multiply, max, matrix multiply, and a nonlinear activation. Each one stores the value it produced and owes its inputs a derivative on the way back. Those two facts are the whole contract.

The upstream gradient says how much the final loss moves when the output of this gate moves. The local gradient says how much that output moves when one input moves. Multiply them and the input has its share.

upstream = dL_dout
local = dout_dx
dL_dx = upstream * local

An operation sees its own local derivative and the signal arriving from the operation after it. Nothing else about the network reaches it. Chain those backward and every parameter gets a gradient.

The forward pass buys memory so the backward pass can exist

The backward pass runs on values the forward pass already computed. A multiply gate needs its original inputs. A ReLU needs the side of zero it landed on, and a softmax loss needs the probabilities it produced.

This is why a neural net library stores activations during the forward pass. The memory is deliberate. It buys the ability to compute gradients later.

Gradient flow has four patterns

Four patterns cover most of the gates you derive by hand. Each pattern comes with the value the gate must cache to run it.

• Add distributes the upstream gradient to both inputs, and caches nothing.
• Multiply sends each input the upstream gradient times the other input, so it caches both.
• Max routes the whole gradient through the winner, so it caches which input won.
• ReLU passes gradient for positive activations and blocks it for negatives, so it caches the sign.

Once you can name the pattern at each gate, the backward pass is local rules chained in reverse. The forward pass decides the routing, and the cache is where it wrote that decision down.

Local derivatives are what make autodiff modular

Give every operation one interface. The forward returns an output and stores what it needs. The backward receives the upstream gradient and returns gradients for its inputs.

Matrix multiply, ReLU, a normalization layer, and a loss all participate on that interface. Each one has to know only its local derivative. A new operation joins an autodiff system without that system knowing anything else about your model.

The library handles gradients by running this interface for you. You still have to know what it saves, routes, multiplies, and sums.

The bugs are boring, which is why they survive

Backprop bugs arrive as a wrong transpose, an omitted batch average, or a forgotten cache. They also arrive as a gradient with the wrong shape, or a regularization term applied to the bias. A wrong broadcast still produces numbers. The numbers land on the wrong weight.

Shape discipline keeps blame arriving at the right parameter. Assert the shape of every gradient against the tensor it corrects. The assertion fails where the bug is.

A gradient that is absent, too large, or exactly zero has a local cause. Look at the operation, the shape, the activation, or the loss connection.

Run a numerical gradient check on a tiny case before you trust a large training run. The optimization note ran that check on the whole loss. Here it points at one gate.

The Builder Test

Trace one scalar by hand. Write the forward values and the loss, then walk backward one operation at a time. Shrink the graph until you can inspect every local derivative by hand.

The trace itself is simple. The hard part is doing it the same way through many gates and keeping the thread. If the manual trace disagrees with the code, trust the trace until you find the bug.

What Carries

Learning is prediction, mistake, blame, correction. Backprop is the blame made precise.

The loss decides the goal, and you chose the loss back in the linear classifier note. Backprop carries the consequence of that goal backward through the network. With blame delivered to every parameter, the open question is what the middle parameters are supposed to represent.

Optimization is repeated local correction. The loss curve is the instrument that names which correction is broken. The height you descend is the loss chosen in the note on linear classifiers. The coordinates are the parameters. Once a score function and a loss exist, training becomes a movement problem.

The job is to move downhill without wasting steps and without walking into nonsense.

The gradient is local advice about a surface you cannot see

A gradient says how the loss changes when you nudge each parameter. It is local advice about the point you stand on. It names the direction that rises fastest right here, and descent steps the other way.

In a real network the parameter space has millions or billions of dimensions. You do not visualize that surface. You probe it, one local measurement of slope at a time.

The gradient never points at the best answer. It points at the steepest local change. It earns its keep when you repeat small corrections in the direction it gives.

Random search is the embarrassment baseline

One crude method samples random weights and keeps the best. It ignores the shape of the loss surface and treats learning as lottery tickets. It must make you uncomfortable.

Run it once anyway, as the floor. It gives the first loss value to beat, which makes the next run comparable. An optimizer that cannot beat random poking is reporting a broken setup. Look at the data path and the loss.

SGD trades an exact gradient for many more updates

Computing the gradient over the full dataset at every step costs too much. Stochastic gradient descent uses minibatches instead, and the estimate it returns is noisy. A full-dataset step is exact and rare. A minibatch step is noisy and frequent, and frequency wins the wall clock.

The noise buys something. It helps the optimizer escape sharp local quirks. It also makes the step size matter more, because a full stride along a rough direction overshoots.

The learning rate decides whether descent is careful, slow, or chaotic

The learning rate is the size of the step you take blindfolded. Too small and training crawls. Too large and the loss jumps or explodes. A stable run moves downhill at a pace the surface supports.

Sweep the learning rate before you change anything else. A run that crawls and a run that explodes both name that knob first.

Read the curve before you touch the architecture

The mistake is to treat gradient descent like a button. Optimization stays empirical even when the math is clean. Your instruments are the loss curve, the gradient norms, the validation behavior, and the update magnitudes.

• The loss is flat from the first step. Examine the learning rate, the gradients, and whether the data reaches the model at all.
• The loss explodes. Examine the step size, the initialization, and the loss for numerical safety.
• Training improves and validation stalls. The model is learning the training set more than the task, so examine overfit and leakage.
• Both improve slowly. The setup can work, and the step size or the capacity is underpowered.

A gradient norm at zero and a loss that will not move are one finding on two instruments. Learning rate, labels, batch size, initialization, and implementation bugs announce themselves here before final accuracy does.

Minibatch noise complicates the reading. The per-step loss is jagged by construction, so a single step is not a reading. Compare a window of steps against the window before it. A jagged curve trending down is doing its job. A jagged curve that stays flat sends you back to the list above.

Numerical gradients are slow truth

A numerical gradient changes one parameter by a tiny amount and watches the loss move. An analytic gradient computes the derivative directly, and training runs on it because it is fast. Numerical gradients are too slow for training. They are the reference that says whether the analytic ones are right.

When you write a new loss or a new layer, check it on a tiny input first. Numerical gradients are affordable there. The same habit has a larger version. Before a long run, make sure the model can overfit a tiny batch. Both checks are cheap, and both stop a broken setup before it costs you hours.

The Builder Test

Predict the shape of the loss curve before you launch the run. Write the prediction down, launch, and compare the first steps against it. Name the mismatch as one of four things: scale, learning rate, data, or gradients.

Sweeping the knob until one curve looks good teaches you nothing about what was broken. The prediction turns a run into evidence. A curve that matches confirms the setup. A curve that misses names the bug.

What Carries

A single gradient can be noisy, local, or badly scaled. The curve over many steps is the evidence.

A gradient helps only when it reaches every parameter. The next note is the bookkeeping that delivers it.

A linear classifier throws away the training set and keeps a weight matrix instead. Nearest neighbor kept every image and changed only the comparison rule. This model reverses that trade, and it is the first real model in the course. The price of the compression is one averaged template per class.

The whole model is one matrix multiply

The function is blunt. Flatten the image into a column, multiply by a weight matrix, add a bias, read one score per class. For CIFAR-10 that is 3,072 pixel inputs and 10 class scores.

scores = W @ x + b
predicted_class = argmax(scores)

The model is simple enough that shape mistakes are embarrassing and common. Write the shapes down once and check the code against them.

x:      3072 x 1
W:      10 x 3072
b:      10 x 1
scores: 10 x 1

If you can write those four lines without looking, you understand the model.

The bias moves the decision boundaries. Without it, every boundary must pass through the origin. A common trick appends a constant 1 to every input vector. Then W absorbs it as one extra column.

That is bookkeeping. Clean bookkeeping matters, because backpropagation will depend on boring and correct shapes.

A row of W is a class template you can reshape into an image

Each row of W is one class template. It likes some pixels, dislikes others, and returns a single number. That number is a dot product between the row and the image.

The row has the same length as the flattened image. Reshape it back into image shape and you can look at what the model learned about that class. In early vision that window matters, because the model stays small enough to inspect all the way down.

Geometry comes before probability here. The classifier cuts pixel space with planes, and each score reports which side of a boundary holds the image. The size of the score reports the strength of that answer.

One template must average every visual mode of the class

One template per class is also the limit. Cars appear red and blue, front-facing and side-facing, and half-occluded. One row of W must average all of those modes, and the average is a blurry compromise.

Those modes are the pressure list from the overview note. Pose, scale, color, and occlusion are what a single averaged row cannot hold at once.

The weights also use whatever separates the training set, including background and scale. A feature that splits the training images without generalizing still lowers the loss. The failure stays legible, which is the useful part. When one boundary cannot separate the visual modes, you know why deeper representations become necessary.

SVM and softmax punish different stories

Multiclass SVM loss asks more than whether the correct class scores highest. It asks whether the correct class beats every rival by a margin.

The margin is the point, because a barely correct score is fragile. A score that wins by enough leaves room for a rival to move.

A rival sitting inside the margin costs loss. Once every rival clears it, that example contributes zero data loss and SVM stops pushing on it.

Softmax starts somewhere else. It turns the same scores into a probability distribution. Cross-entropy then charges the model when the true class gets low probability.

Softmax also hands you a number that looks like confidence. High probability and calibrated probability are separate claims, and the held-out split settles which one you have.

SVM keeps rivals away from the correct class. Softmax piles probability mass onto it. Both make bad scores expensive, and they press on the same ten numbers in different places.

The difference shows up in the training curves. SVM can ignore examples that are already safe. Softmax keeps shaping confidence long after the ranking is correct.

Accuracy counts whether the top label matched. Loss reports how strongly the model was wrong or right, and the optimization note reads that curve.

Regularization states which fitting solution you meant

Without regularization, many weight settings fit the training data, some sharp and brittle, others smaller and smoother. Regularization names which one you prefer.

L2 penalizes large weights, which stops one accidental pixel from carrying the decision. The data loss says fit the labels. The regularizer says fit them in this kind of way. That preference is the model you meant to build.

The Builder Test

Run three checks before you reach for a bigger model.

• Reshape each row of W into image shape and look at it.
• Inspect the class confusions and ask which two templates overlap.
• Compare train loss with validation loss on the same run.

Read the gap between those two numbers. A wide gap means the model fit noise instead of structure.

What Carries

Scores become behavior only through a loss. The loss defines what the model pays attention to, so choose it with care. It says how strongly the model is wrong. The next question is how the parameters move because of it.

Image classification is a promise that the rule holds on the next image. The split is the only instrument that tests the promise. The overview note listed evaluation as a stage of the pipeline. The split is that stage doing its work.

A 32 by 32 RGB image arrives as 3,072 numbers

A larger image arrives as a bigger slab of the same values. The first job is to turn those numbers into one label from a fixed set. Understanding is not in the slab.

You cannot hand-write the rules for cat or car in pixels. The same object appears across poses, backgrounds, lighting conditions, scales, textures, and partial views. Every condition adds cases, and the rulebook explodes before it covers one class.

You collect examples because you cannot write the rulebook

So you collect examples. Give the model labeled images, pick a model family, and learn from the examples. Then measure on images the model never saw. A model that succeeds only on the images it memorized is a lookup table with good manners.

Every classifier in this course keeps the same contract.

model.train(X_train, y_train)
y_pred = model.predict(X_test)
accuracy = mean(y_pred == y_test)

Train on labeled images, predict on unseen images, score the match. The implementation behind train and predict changes with every model that follows. The contract stays fixed, which is why the crude baseline is still worth writing.

Nearest neighbor is the honest baseline because it makes you define distance

To train, nearest neighbor remembers every image. To predict, it compares the new image against every stored image and copies the closest label. It forces you to define distance before it can answer anything.

L1 adds the absolute pixel differences. L2 squares them, which punishes large disagreements harder. Neither distance understands an object, and both compare numbers.

The first neighbor can be an outlier. k-nearest neighbor asks the top k neighbors to vote, which smooths small islands of bad decisions. Do not trust the first kid who points at an answer. Ask five nearby and take the vote.

Nearest neighbor pays at prediction time. It stores the whole training set and scans it for every query. That cost grows with the dataset. Whatever this baseline already solves is not evidence for the model that replaces it.

Pixel closeness is not semantic closeness

Move a cat a few pixels to the right and the raw vector changes a lot. The picture still means the same thing to you. Change the background behind a car and the model finds a closer match in the wrong class. The colors line up and the label does not.

You stay invariant to pose, lighting, small translation, partial occlusion, and background clutter. Pixel distance treats each of those as a large change. The overview note named that list as the standing pressure on every vision model. Nearest neighbor fails all of it at once.

A vision system must learn those invariances. It cannot assume them. The repair has to come from the representation, because L1 and L2 read only the numbers they are handed.

Tune on the test set and it becomes training data

Train on the training set. Tune k, the distance metric, the learning rate, and regularization on the validation set. Touch the test set once, at the end. Validation is the wall between exploration and final judgment.

A clean split tells you whether the model learned the object, the dataset, or a shortcut. A dirty one tells you nothing and reads the same.

The split leaks in two ways. Near-duplicate images cross from training into validation, and the validation number then reports memory. Repeated peeking at the test set carries the same information across one decision at a time.

Each leak ends in the same place. The test number stays clean and lies. Peek often enough and you optimize the scoreboard instead of the task. The split is the first honest boundary between memory and vision.

The Builder Test

Sort the misclassified images by confidence and start at the top. Inspect those confident mistakes by category, by background, and by distance to their nearest neighbors. Read the confusion matrix and find the class pairs carrying the real ambiguity.

Some of those mistakes are label noise, and a wrong label is not a model failure. Audit the split before you believe any number. Search for near-duplicates crossing training and validation. Check class frequency and photographer bias in the classes you confuse most. If a shortcut explains the confident mistakes, write the test that falsifies it.

What Carries

The visible label is cat, car, truck, or bird. The hidden label is whether the rule survives outside the training set. Nearest neighbor keeps the data and only changes the comparison rule. The next model throws the data away. It keeps parameters instead.

Every method in this sequence answers one question: what representation survives an image that nobody arranged for the model. The notes run from pixels to systems that classify, localize, generate, attend, transfer, and fail in ways you can name.

An image is a block of numbers until you build the machine that can use it. From those numbers the model must recover objects, shapes, texture, motion, and meaning. The work is turning raw arrays into decisions you can act on.

The pipeline is what you are learning, and it has seven stages

The pipeline is the lesson. The stages run in this order: data, representation, loss, optimization, architecture, evaluation, and failure analysis. Each stage decides what the next stage can see.

Data sets the ceiling on what the model can learn about. Representation decides what survives the first transform. The loss states what counts as wrong, and the model optimizes that statement rather than your intent. Optimization moves the parameters, and backpropagation is the bookkeeping for blame that tells each one how it contributed. Architecture makes some structure cheap to express and leaves the rest expensive.

Evaluation is the stage where the split does the work. Train fits the parameters. Validation picks the choices you make by hand. Test reports what happens on data you never touched. Failure analysis then decides what you fix next, and the fix often lands back at data or representation.

The pressures are the same at every stage: pose, lighting, scale, background, occlusion, viewpoint, and class imbalance. Every later note tests against that one list.

Every method asks the same question in a different costume

The sequence defines the task first, then makes the simplest models fail in public. After that come gradients, backpropagation, neural networks, convolution, and the training discipline that survives contact with real data. Later notes add attention, transformers, generative models, NeRF, and adversarial examples.

Each method answers the same two things: what must stay fixed, and what the model must learn from data.

Nearest neighbor fixes the data and changes the comparison rule. Linear classifiers learn a weight matrix, and from there the parameters carry the model. Neural networks learn layers of representation. ConvNets learn local filters and share them across space.

Transformers learn routing. Generative models learn the data distribution. Adversarial examples show that the learned representation is not the same thing as human vision.

Each answer buys an invariance and pays for it with information it throws away. Convolution buys locality and weight sharing because images repeat local patterns, and it gives up per-pixel parameters.

Historical order teaches the names. Shortcut order teaches the work: what shortcut each architecture blocks, and what new shortcut it opens. Track what each method makes easier to represent and which failure it leaves exposed.

The contract holds even when the architecture changes

Every note in the sequence works under one contract. You start with raw visual data, define a task, choose a representation and an objective, and optimize it. Then you show the result survives new examples.

That is why notes written before transformers still decide how I read the new ones. Six standing checks carry across all of them.

• Name the input and its exact shape.
• Say where the labels came from and who made them.
• Write the loss down and read it as the goal the model pursues.
• Pick the baseline that makes any gain believable.
• State what changed after training.
• Find what breaks under distribution shift.

Read every note with one builder question

One filter decides what I keep from a note. If I must implement this or debug it tomorrow, what do I need to remember?

That filter cuts a lot of academic fog. It keeps the shape math, the loss intuition, and the failure modes. It keeps the small implementation details that decide whether a notebook trains or only looks like it trains.

A clean accuracy number is the easiest thing to trust too early

A model can win the average case and break under a lighting change, an occlusion, or a rare class. It can also break because it found a background cue and used it.

Vision models see tensors and correlations, and a background cue is a correlation like any other. That is why baselines, augmentation, saliency, ablations, and error analysis carry the weight here. Trust an accuracy number after it holds on a slice you chose to be hard.

The Builder Test

Pick one misclassified image and explain the mistake. Name the feature the model used and the evidence that proves it. Then name the one change that makes the model less brittle.

You pass when another person looks at the same evidence and sees the same cue.

What Carries

Generalization is the whole point. Training accuracy is a rehearsal, and the image the model never saw is the performance.

The first stage fixes the task: pixels in, one label out. Start there. The label must hold under the pressure list, and the split is the instrument that reports whether it does.