Exact Likelihood Has a Price in Order and Speed

An autoregressive image model buys an exact likelihood. It pays twice: once in the order it commits to, once in the sampling speed it gives up. The mask is what keeps that contract honest.

PixelRNN models an image as a sequence. Predict the next pixel from the pixels before it, then repeat. A model that assigns high probability to real images learned something about the image distribution.

The generative models note named this family and stopped at the price. Two things set that price: a product of conditionals, and a mask that enforces the order.

Factor the image into conditionals and the objective stops being vague

An autoregressive model decomposes the probability of an image into a product of conditional probabilities. The model predicts each pixel from the pixels that came before it. Train it by maximizing likelihood, or by minimizing negative log likelihood. Those are the same objective written two ways.

The objective is explicit, which is cleaner than many generative methods manage. You know what you optimize. The number the model reports is the number the training loop moved.

The order is part of the model

Choose the order and you choose the prediction problem. Each pixel inherits one context, the pixels ranked before it, and nothing else. Change the ranking and every conditional answers a different question.

Every prediction carries a context and a probability. The visualizing ConvNets note left confidence as a claim about the model itself. Here the likelihood states the context it assumed and the quantity it computed. A general sense of image quality decides nothing.

Masked convolutions stop the model from seeing the pixel it must predict

PixelCNN uses masked convolutions so each prediction sees only the pixels the order allows. A convolution kernel covers a neighborhood in every direction. The mask zeroes the taps at the target position and after it. Causality then holds inside a convolutional architecture.

Drop the mask and the model reads the answer it must predict. Training loss falls for the wrong reason. The note on setup failures asked whether the train-time and test-time paths agree. A leaking mask breaks that agreement inside one forward pass, because at sampling time those pixels do not exist.

The mask also makes training parallel. A recurrent scan walks positions one after another. Masked convolutions score every position of a training image in one pass, and the autoregressive factorization survives.

Sampling pays the bill for exactness

The bill arrives at generation time. Every pixel depends on the pixels before it, so sampling runs one position at a time. An image takes as many forward passes as it has pixels.

Generate all pixels at once and the cost disappears along with the model. That breaks the contract. You trained each conditional with the earlier pixels in hand, and independent sampling removes them.

Global structure must travel through many conditional decisions

Local consistency comes free. Each prediction conditions on nearby pixels that are already fixed, so edges line up and texture stays continuous.

Global structure has to travel. A commitment made in one corner of the image reaches the far corner through a long chain of conditionals. Every link in that chain can drop it.

That is one reason later systems add stronger global representations: latent variables, attention, or diffusion dynamics. Each of those gives long-range structure a route that skips the pixel-by-pixel chain.

The Builder Test

Take one output position. Draw the receptive field around it and mark every input pixel the mask allows through. Then look at the target position and everything after it in your chosen order.

If any of those taps survive the mask, the model trains on the answer. Fix the mask before you read another loss curve.

What Carries

Diffusion and transformer-style token models run most image generation now. The lesson outlives the architecture. A generative model is accountable when its factorization, its objective, and its sampling procedure are all stated and all agree.

An explicit contract like this one is easy to judge. The harder job comes first: name the contract the system needs, then choose the family that can sign it.