Generative Models Learn the Shape of the Data

Each generative family buys one property and pays in a fixed direction. Name the metric first, and the family follows.

The linear-classifier note treated the loss as the real decision, because the loss decides what a model attends to. Generation applies the same discipline to the metric. Until you name one, sample quality stays a matter of taste.

Generation asks which images can come from this data at all

Classification asks which label fits this image, and a classifier answers with one accuracy number. Generation has no such number to stand behind.

A generative model has to put samples on the page. The batch shows what the model treats as typical. It also shows which modes it misses, which details it blurs, and which artifacts it repeats.

Autoregressive models buy clean likelihood and pay in sampling speed

PixelRNN and PixelCNN model an image as a sequence of conditional predictions. The model predicts each pixel from the pixels before it, in one fixed order. That order gives a likelihood you can compute directly.

Sampling has to follow that order, one pixel at a time. The cost lands on generation, and it lands there again on every sample you draw.

Autoencoders compress, and VAEs make the latent space sampleable

An autoencoder maps an input to a latent code and rebuilds the input from that code. The code keeps the information reconstruction needs and drops the rest. Sampling from that space is a separate job, and a plain autoencoder does not organize it for you.

A variational autoencoder puts a distribution over the latents. The encoder predicts that distribution, and the decoder rebuilds from samples of it.

The loss is a negotiated settlement between two terms. The reconstruction term wants faithful outputs. The KL term wants the latent distribution to stay close to a simple prior.

Press the prior too hard and reconstructions suffer. Press reconstruction too hard and the space stops being worth sampling. Most VAEs settle where the samples blur.

The check is cheap. Decode two nearby latent points and look at both images. If the pair stays sensible, the space is organized enough to sample from.

GANs sharpen samples and destabilize training

A GAN runs a generator against a discriminator. The generator makes images, the discriminator separates generated images from real ones, and the generator improves by fooling it.

Sharpness comes out of that game. The discriminator rejects unrealistic samples, so the generator learns from a signal closer to perception.

The same game breaks the training. The generator can exploit the discriminator instead of covering the data, or collapse onto a few modes. The signal keeps moving while the model learns, which is why GAN evaluation stays hard.

Likelihood and sample quality are different scores

One model assigns good likelihood and still produces blurry samples. Another produces sharp samples and makes likelihood hard to compute. Rank the same models by one score and then the other, and the orders disagree.

Comparing autoregressive models, VAEs, GANs, and diffusion-style methods needs a named metric. The candidates are likelihood, perceptual quality, diversity, controllability, and speed.

Each metric stays quiet about its own failure. Likelihood says nothing about blur, and perceptual quality says nothing about missing modes or memorization.

Generation turns into copying when the data, the objective, or the evaluation lets memorization pass as sampling. The classification note named leakage and near-duplicate checks for the same reason. Both ask what the model kept.

A sharp sample is the model being confident about the data distribution. The visualization note put that warning on a class score, and it carries over here. Confidence is confidence under the representation the model learned.

The Builder Test

Take a model you trained and generate a batch. Judge the batch on the one metric you named before you looked at it. Then write down the failure that metric cannot see.

The likely failures are already known. Autoregressive sampling runs slow and VAE reconstructions blur. GAN training drops modes, and diffusion needs a run of denoising steps for every image it makes.

The batch then needs a second check, chosen against the failure you wrote down. Count distinct modes when the metric rewards sharpness. Compare each sample against its nearest training image when the metric rewards fidelity.

If the batch fails that second check, the score it earned does not mean anything yet.

What Carries

Samples are the honest surface of a generative model. One good sample is the cheapest thing that surface can give, and a collapsed model still produces one. Evidence starts at a batch, judged on the metric you named before generating.

Every family here learns a distribution over images. The next representation learns one scene, with rendering supplying the label.