A reward of 1 for a correct answer and 0 for everything else leaves most gradients at zero. If the policy almost never lands a correct answer, the run barely moves. The repair is a baseline: sample the same prompt several times and center the rewards on the group mean. That is what turns an RL update into something you can read.

The goal of RL for a language model is small. Make the model pick answers that score well. The rewards are sparse and the variance is high. Inference is the expensive part.

Sparse rewards leave most gradients at zero

The state S is the prompt plus the response so far. Each generated token joins the state. An action picks the next token. With outcome rewards, treat the whole response as one action A.

The reward scores that whole response instead of each step inside it. Make it verifiable and deterministic. A program parses the final answer and compares it against a ground-truth key. Correct scores 1, and everything else scores 0.

Transitions here are known and dull. Each one appends the token the model chose. Robotics takes its transitions from the world, and they can be unknown.

Many robot states are physically impossible. A language model can reach every token sequence, so the hard part is finding the ones that earn a reward. The same freedom lets the model build scratchpads and chains of thought on the way there.

The policy π is a distribution over next tokens given the current state. Most runs start from a pretrained model and fine-tune it. A rollout starts from a prompt, samples a full response from π, and scores it once.

The objective is the expected reward.

S comes from the prompts, and you sample A from πθ. The gradient of that objective has one term.

One stochastic step follows directly.

• Sample a prompt S.
• Sample a response A from πθ.
• Compute the reward R(S,A).
• Update with R(S,A) times the gradient of log πθ(A|S).

The shape matches supervised learning with two changes. The label A comes from the model itself. The reward scales the update, so a better response pushes harder.

With a 0/1 reward, every wrong sample multiplies its gradient by zero. Only correct samples move the parameters.

A learned reward model changes the shape of the problem. RLHF trains one on preference data, and it returns a real-valued score for any response. The signal is smoother, so the same algorithm behaves differently and needs different tuning.

A baseline cuts the variance and leaves the expectation alone

Naive policy gradient is unbiased and noisy. You want the same expected gradient with less noise. Subtract a baseline B that depends only on the state.

For a fixed S, the baseline term sums to zero across actions. The expectation is untouched and only the variance moves.

Two states and two actions show the effect.

S1,A1 → 11, S1,A2 → 9
S2,A1 → 0,  S2,A2 → 2

Draw the single sample (S1,A2) with reward 9 and the update pushes toward A2. A1 is the better action in that state. The push is noise, and its size comes straight from the raw reward.

Now set B(S1) = 10 and B(S2) = 1.

S1,A1: 11-10 = 1,  S1,A2: 9-10 = -1
S2,A1: 0-1 = -1,   S2,A2: 2-1 = 1

The better action in each state carries +1 and the worse one carries -1. The effective rewards are smaller and centered, and the updates get less noisy.

Each baseline here sits at the average reward for its own state. That is the common choice, and it is the one that strips the most noise.

Name that quantity and the standard identities follow.

With outcome rewards on a full response, Q matches R for the response you sampled. R - V(S) is then an advantage estimate, which is what every method in this family builds.

Most of the family fits one form.

Δ is what changes between methods.

• Naive: Δ = R.
• Baseline: Δ = R - B(S).
• Advantage-based: Δ approximates A(S,A).
• GRPO-style: Δ is centered and normalized inside a group, usually with clipped ratios and an optional KL term.

GRPO gets its baseline by sampling the same prompt again

Estimating V(S) usually costs a second model. A language model offers a cheaper route, because you can reset to the same prompt and sample many answers. GRPO is a PPO-style method that builds its baseline out of that.

For one prompt, collect the rewards R1 through RK. Scoring a whole batch gives a reward matrix shaped [batch, num_samples]. The baseline is the mean inside each prompt.

Center each reward on it.

A response above its group average gets a positive Δ and a push up. A response below it gets a negative Δ and a push down. The group grades itself.

Divide by the group standard deviation to fix the scale.

The note on verifiable rewards objected to this divisor. The case for keeping it is narrow. The update size stops tracking the reward scale, which helps stability. In small runs the centered and normalized versions land in about the same place.

The group baseline brings its own failure mode. If every sample for a prompt earns the same reward, every Δ is zero and that prompt gives no gradient. A sparse reward and a weak policy produce exactly that.

A denser reward buys speed and sells loopholes

RL optimizes what you can measure, so the reward function decides what the model becomes. The toy that exposes this is sorting. The prompt is a fixed-length list of n numbers, and the model must return the same numbers in sorted order. Prompt length and response length are fixed.

The first reward is exact match. Score 1 when the output equals the sorted sequence and 0 otherwise. It matches the goal exactly, and a random policy almost always scores 0, so the run never starts.

The second reward counts positions that match the sorted truth.

truth: 0 1 2 3
0 1 2 3 → 4
2 1 3 0 → 1
1 0 2 3 → 1

That reward produces a gradient from almost any output. It also hands the same score to two wrong answers with nothing in common. The model learns that one wrong answer is worth as much as the other.

The third reward adds inclusion and adjacency. Inclusion counts how many prompt tokens appear anywhere in the response. Adjacency counts how many neighboring pairs sit in sorted order. The signal gets denser, and some patterns score well without sorting anything.

Denser rewards keep the run moving, and they pull the model toward easy wrong strategies that pay. Reward design sits between those two failures and stays fragile in both directions.

One patch keeps Δ only for the top-scoring responses in each group and zeroes the rest. It slows the drift into mediocre partial-credit modes. It also changes the signal you optimize, so measure it before and after you turn it on.

The ratio collapses to one unless you detach the old policy

In code the objective becomes a loss.

Here log πθ(A|S) is the sum of the per-token log probabilities for that response. Run the model for logits, take log_softmax, then gather the log probability at each sampled index. The result is shaped [batch, num_samples, positions].

GRPO takes several gradient steps on the same sampled responses, and reuse is how you amortize the sampling cost. The policy that produced those responses goes stale in the meantime. PPO-style methods correct for that with a ratio against a snapshot.

Clipping keeps a reused batch from moving the policy too far.

• Compute r for each response.
• Clip it to the band from 1-ε to 1+ε.
• Take the minimum of rΔ and clipped_rΔ, then flip the sign for a minimization loss.

A KL term pulls the policy back toward a reference model.

KL is the expectation of log(P/Q) under P. The direct estimate of it is noisy, so many implementations use an unbiased form with lower variance.

Compute it per token, then average over the batch, the samples, and the positions.

The loop that holds all of this is short.

• Sample prompts, then sample several responses per prompt from the current policy or a frozen snapshot.
• Score every response, then compute Δ with the raw, centered, or normalized scheme.
• Compute logp_old for those responses, store it, and detach it.
• Compute log probs under π_ref when you use the KL term.
• Take several gradient steps on the same responses without resampling.
• Inside a step, recompute logp_current, form the ratios, clip them, and apply Δ.
• Add the KL term when you use it, then backprop and update θ.
• Refresh π_old to the current policy for the next batch, and update π_ref less often when it must move slowly.

Every update changes πθ, so the data distribution moves under you for the whole run. The loss curve is a weaker signal here than in supervised training. Read the reward on held-out prompts instead.

Outcome rewards assign credit bluntly

One reward covers the whole response, so the same Δ multiplies the log probability at every position. A response that nails one pivotal token and then drifts gets the same push on every token. Nothing in the method knows which token did the work.

Process rewards score the intermediate steps and can fix this. They are hard to design for language reasoning. Until you have one, every token in a rewarded response gets credit by association.

The cost is inference and model copies

Inference dominates the bill. Every prompt needs K full generations before one gradient step, and the policy must produce all of them.

The system also holds more than one model.

• The policy you update.
• A reward system, which is a verifiable checker or a trained reward model.
• An old snapshot for the ratios, or the stored old log probabilities in its place.
• A reference policy for the KL term.
• A critic or value model, in the methods that use one.

Full copies cost memory, which is why storing the old log probabilities is worth the bookkeeping. You also need distributed sampling, distributed reward computation, model synchronization across workers, and careful variance control.

That combination makes an RL run harder to operate than supervised pretraining. This unit prices a run before it starts. Price this one in sampled tokens, not in parameters.

The Builder Test

Pick one prompt with a checkable answer. Sample K responses from your current policy and score them all.

Compute Δ twice. Once with the raw rewards, once with the rewards centered on the group mean. Count how many responses carry a nonzero Δ in each version.

With a 0/1 reward, the raw version leaves every wrong response at zero. The centered version gives the wrong ones a negative push, and the count of usable deltas jumps.

Then run the same check on a prompt the policy already solves. Every sample comes back correct, the group mean sits at 1, and every centered Δ is zero. That decides which prompts are worth sampling. A group that agrees teaches nothing.

Name the proxy before you trust any improvement. Write down three things.

• What the reward pays for.
• What the reward ignores.
• The pattern that scores well without solving the task.

A model learns the shape of a helpful answer or the length a reward prefers without getting more reliable. If the proxy is incomplete, stronger optimization makes the model better at the proxy and worse at the goal. The evaluation note asked the same of a benchmark score. A reward is a benchmark you optimize against on purpose.

What Carries

One habit carries out of this unit. Name the binding constraint before you reach for a technique, then price it.

Here the constraint was variance under a sparse reward. The price was K samples of every prompt, paid in inference before one gradient step moved.

Each stage priced the one after it, from the tokenizer forward. The reward is the last stage in that chain, and it decides the behavior you get.

Every number here answers three questions: what produced it, what it cost, and how you catch it lying. Before your next run, write down what one gradient step costs you in sampled tokens. If you cannot write that number, do not start the run.

Push a proxy reward hard enough and the true human win rate turns down. The proxy score keeps climbing the whole time. That curve is why the reasoning systems that work now optimize rewards a program can check.

Preference optimization makes two decisions, and both belong to you. You decide what counts as good. You decide how hard to push on it.

Push the proxy and the true win rate turns down

The setup is pairwise preference data. A human sees two model responses to a prompt and picks the better one. A reward model learns from those picks, and the policy maximizes that reward. The reward model is the proxy, and the rest is a question of how far to trust it.

DPO removes the separate reward model. It writes the reward as a log ratio between the target policy and a reference policy. That ratio goes into a Bradley-Terry preference model, and training becomes supervised. The update raises the likelihood of the preferred response more than the dispreferred one. It grows when the implied reward disagrees with the human pick.

Raise the optimization pressure on that proxy and two curves separate. The proxy reward climbs without limit. The true human win rate climbs, turns, and falls. In that gap the policy is fitting the reward model, and the preference behind it has stopped moving.

Preference labels are noisy. Human preferences are complex and incomplete, so the reward model learns a partial picture. The optimizer finds the part of that picture that is wrong and lives there.

Students ran RLHF with three reward sources. They used real human preferences, noisy AI feedback, and clean noiseless AI feedback. The curve appeared for human preferences and for noisy AI feedback. Clean noiseless rewards almost erased it. The noise produces the turn.

I have stopped reading a rising reward curve as progress. It reports the proxy, and the proxy is the thing under attack.

Pretraining and supervised fine-tuning fit likelihoods, so the model keeps a probabilistic view of the world. RLHF optimizes a reward. In experiments the RLHF models became overconfident, most visibly at temperature 1. They scored worse on calibration than the SFT baselines.

Calibration is absent from the reward, so nothing in the objective protects it. Stop treating an RLHF model as a well-calibrated probability estimator. If a downstream system reads probabilities out of the model, keep an SFT checkpoint for that job.

The preference stage is also worth less than the first papers suggested. AI2 measured PPO ahead of DPO. Later work, Tulu 3 included, did not reproduce that ordering. A better supervised stage gave most of the gain either method gave. In one experiment SFT alone matched RLHF, and only length-normalized DPO added a small gain.

Results move with the base model, the data, and the environment. One paper does not settle the ordering.

Verifiable rewards remove the noise that caused the failure

Human approval is noisy, easy to game, expensive, and prone to overoptimization. The classic RL successes ran on rewards that were stable, cheap, and scalable. Keep the RL machinery and change the task.

Math, coding, logic, and puzzles have clear correct answers or strong automatic judges. A program decides the reward, and the label noise that bent the curve goes away.

The scaling-law note bought an expensive decision with cheap measurements. A verifier runs the same trade for reward. It costs an answer key or a unit test, and it returns the same verdict every time.

This change is the base of the o1-style models. DeepSeek R1, Kimi K1.5, and Qwen 3 all sit on it.

A verifier removes the noise inside the proxy and leaves the proxy standing. The distance between a correct answer and a good answer stays open. Clean verifiers cover a small part of what people ask a model to do. Every one of those pipelines still ends in RLHF for the rest.

PPO works and costs a value model the size of the policy

Policy gradient maximizes expected reward under the policy. The gradient raises the probability of the high-reward samples. It lowers the low-reward ones.

The naive version is fully on-policy. Every update needs fresh rollouts and fresh reward computations. For a language model the rollout is the expensive part. You want several optimization steps out of each batch of rollouts.

TRPO reuses samples from an older policy through importance sampling. Likelihood ratios correct the samples. A KL constraint holds the new policy near the old one and blocks very large steps.

PPO replaces that constraint with clipping. It multiplies the likelihood ratio by an advantage term. Then it clips the ratio to a band such as [1 - ε, 1 + ε]. The clip removes any gain outside the band, so one update cannot move the policy far.

PPO also carries a value network that predicts expected returns. Generalized advantage estimation lowers the variance of that estimate. Two more parameters, γ and λ, set the balance between bias and variance.

For a language model the setting is a contextual bandit. The prompt is the input. The action is one full generated sequence. One scalar reward arrives at the end, and the task has no real state transitions.

Training still runs at token level. The task reward lands at the end and reads as correct or incorrect. The loop broadcasts that final scalar backward across the tokens. Per-token KL penalties hold the policy near a reference policy.

The outer loop is ordinary. Collect rollouts, compute rewards and advantages, take a clipped policy step, then clip the gradients. The cost sits in the parts list.

• a reward model
• a value model about as large as the policy
• a GAE implementation with its own tuning
• KL penalties for each token or for each sequence
• many small tricks that decide whether the run stays stable

The run then holds the policy, a reference copy for the KL penalty, a reward model, and a value model. The value model alone doubles the memory and makes the training harder.

PPO is clean on the page. Every version that survives contact with a large model carries that parts list, and GRPO exists to cut it.

GRPO drops the value model and uses the group as the baseline

GRPO keeps the PPO-style clipped objective and removes the two most expensive parts, the value model and GAE. For each prompt you sample a group of G responses. Each response gets a scalar reward.

The group mean and the group standard deviation turn that reward into an advantage.

That advantage goes into the clipped objective, and no value network appears anywhere in the loop.

The group is every response to the same question. The group mean is then a baseline that carries the difficulty of that question. Subtract it and the variance drops. Learning depends on relative success inside one prompt, and the standard deviation normalizes the scale across prompts.

The baseline arrives for free in parameters and gets paid for in rollouts. You sample G responses for every prompt, so the cost moves out of a second network and into generation.

GRPO can add clipping or KL terms to hold the policy near a reference. In a fully online single-step configuration you can drop the clipping. You are left with a policy gradient and a baseline.

DeepSeekMath is where GRPO earned the default position on math. It ran two reward types. An outcome reward pays 1 for a correct final answer and 0 otherwise. A process reward model scores the reasoning steps instead.

GRPO with outcome rewards beat fine-tuning on correct outputs. It also beat the online variant that refreshes those outputs from the current model.

The original GRPO objective pushes chains of thought longer for the wrong reason

You can subtract any baseline that does not depend on the action. The policy gradient stays unbiased and the variance usually drops. Division is a different operation, and the theorem does not cover it.

Dividing by the group standard deviation changes the gradient. The update no longer matches the true policy gradient, and the damage shows when that standard deviation is small.

The standard deviation goes small when a group agrees with itself. Every answer is correct on an easy problem. Every answer is wrong on a hard one. Division inflates those advantages, and training drifts toward both extremes. The mid-difficulty problems carry the learning signal, and those are the ones the update walks away from.

The original objective also divides the total reward by the output length. On a wrong answer the reward sits near zero and the KL penalties are negative. A longer sequence spreads that negative number across more tokens and lowers the per-token penalty. The model learns to fail at length.

On correct answers the same normalization pushes toward shorter outputs. A model that is stuck takes the other branch and runs a long pathological chain of thought.

Dr-GRPO removes the division by the standard deviation and removes the length normalization from the reward. Accuracy held or improved on tasks such as GSM8K. The uncontrolled length growth stopped, and the output length settled at a reasonable plateau.

That plateau is the evidence. Some of the very long chains in GRPO runs are artifacts of the objective. High performance does not need them.

R1-zero is where the length story started. It begins from DeepSeek V3 before instruction tuning and before RLHF, with no chain-of-thought SFT. It runs RL with verifiable rewards, mostly on math. A binary accuracy reward scores the answer, and a format reward keeps the reasoning inside think tags.

The format reward is doing quiet work. It keeps the reasoning contained and machine-readable, which is what makes the rest of the pipeline possible.

As RL continues the chains get longer. The paper reads that as learning to think longer on harder problems. It reports aha-style behavior such as backtracking. After Dr-GRPO that reading needs a discount.

A length-biased objective produces some of the same growth. Large base models produce aha-style text from SFT alone. R1-zero still shows the real result. Outcome-based RL on verifiable math rewards turns a base model into a strong reasoner.

If your GRPO run grows chains without an accuracy gain, audit the objective before you credit the model.

The evaluation note named length bias as one of two things that decide a leaderboard without appearing on it. The post-training note showed every shortcut in the feedback taught back as behavior. Here the same bias arrives through the objective, which is the third door into one failure.

R1, Kimi, and Qwen converge on one recipe

Three systems from the same window landed on the same four stages. Long chain-of-thought SFT comes first. Then RL on verifiable outcomes, then general RLHF, then distillation into smaller models.

R1 aims at production use across tasks. It starts from DeepSeek V3 and fine-tunes on many long reasoning traces. The source of those traces is not stated, and distillation from strong models is the likely answer. The goal is a model that produces long structured reasoning easily and stays readable through RL.

Even a small amount of chain-of-thought data raises math scores for base models such as Qwen 2.5.

The reasoning RL stage repeats R1-zero with two additions. Format rewards enforce the thinking tags. A language consistency reward stops the chain of thought from mixing languages at random.

After reasoning RL the model is strong on math and logic and less friendly. They run SFT on mixed reasoning and nonreasoning tasks. Then they run GRPO-like RL on preference data, which obeys the DeepSeek V3 pipeline.

Distillation closes the pipeline. R1 generates traces and answers, and smaller models train on them. Those distilled models improve on math benchmarks against base models of the same size.

Kimi K1.5 arrived at about the same time and matches or beats o1 on many reasoning benchmarks. Its edge comes from the data work. They tag math questions by domain and balance the domains. They drop multiple-choice and true/false items, which are easy to guess or game. They keep questions a program can check, which means short text answers, numbers, and code outputs.

Difficulty selection does the rest. They run the base SFT model on each problem without reasoning and sample several answers. They keep the problems that stay hard, for example the ones that fail best-of-8. RL then spends its budget where the model is weak.

A chain-of-thought SFT step comes first here too. The data and prompt details are thin in the paper. The goal is a warm start that already produces usable reasoning traces.

Their objective starts from expected reward with KL regularization to the base policy. A nonparametric trick like the one in DPO writes the implied rewards as policy ratios. They then penalize the squared difference between the implied rewards and the actual rewards. The gradient looks like a policy gradient with a batch-mean baseline and an explicit squared KL term.

Qwen 3 runs the same pipeline and adds discipline to the data. They decontaminate against the validation benchmarks and filter the SFT data by hand. That pass removes low-quality chains and guessing-style chains. Their reasoning RL uses about 3,995 examples and still produces strong gains. On a strong base model, RL on verifiable tasks is data-efficient.

Chain-of-thought length is now a design parameter

The inference note priced generation in bytes moved, since every sequence drags its own KV cache. A long chain of thought is that bill in a new form. Kimi treats length as a control variable and pays for it directly.

RL on language models is system-heavy for the same reason. Rollout generation and training updates run at once, and rollouts are slow. Weights move from the RL workers to the inference workers, and trajectories move back. Long chains widen the length variance and make batching worse.

In each batch they find the shortest output and the longest output, then place every output on that range. A correct answer gets a length reward that pulls toward the short end. A wrong answer gets pulled toward the middle, away from both very short and very long. Correct solutions compress, and long wrong solutions earn nothing.

Length pressure applied early collapses the model into short wrong answers. The schedule has two phases. The model first learns to solve the problem. The stronger length reward arrives after that and compresses the reasoning while accuracy holds. With length control the average chain grows early, then flattens.

Qwen 3 moves the control to inference. One model carries two modes. The think mode produces a chain of thought before the answer, and the no_think mode answers directly. Special tokens in the prompt select the mode, and the SFT data shows both behaviors.

Forcing a stop during training teaches the model to switch from thinking to answering. At inference you set a budget on thinking tokens and force the answer when the budget runs out. Test-time scaling is smooth in both directions. More thinking tokens raise accuracy, and fewer lower it slowly with no cliff.

Set that budget per request and price it like any other serving cost.

The stages pull against each other. Reasoning RL raises math and STEM scores. Thinking mode fusion keeps those gains and adds the control. General RLHF improves instruction following and general tasks. It costs a small amount of thinking-mode math and STEM performance.

That is the price of a friendly generalist next to a reasoning specialist. You pick which one you are shipping. Data curation, reward design, and inference cost now matter as much as the choice of RL algorithm.

The Builder Test

Run the reward past the point where the score still improves. Then measure the thing you wanted, on held-out cases the reward never touched. If the score keeps rising while that measure falls, you have found the turn. Finding it on purpose costs one run.

Before you trust an improvement, name the proxy out loud. Say what the reward paid for and what it ignored. Then find the cheapest way to raise that score without doing the work. The optimizer will find that path too.

Supervised examples teach format. Preference methods and verifier signals teach which answers to make more likely. A model learns the shape of a helpful answer, the length a reward prefers, and the refusal pattern. Reliability does not follow.

What Carries

Reward design is model design. A reward a program can check is a reward you can audit after the run. A reward that only a human panel can judge drifts, and the optimizer finds the drift first.

Alignment spans data design, objective design, evaluation, the refusal and usefulness trade-off, monitoring, and rollback. One fine-tuning run is one item on that list.

Post-training teaches the model what the system rewards. Whether that matches what you meant is a separate measurement, and you must run it.

PPO, GRPO, and Dr-GRPO all rest on one estimator of the policy gradient. The next note takes that estimator apart.

Post-training decides how capability shows up in a product. Pre-training loads that capability into the weights and stops there. The same base model can come out helpful or harmful, depending on the steps that follow.

Every shortcut in the feedback becomes behavior. A rater with one minute for each pair rewards confident structure. The model trained on those labels learns to write confident structure. Nobody put that rule in the guidelines.

Pre-training packs capability and post-training decides when it appears

Pre-training runs over web text, books, and code. The model picks up language, reasoning patterns, coding, and many world facts. It does not come out of that behaving like an assistant. The model must obey instructions, stay helpful, stay safe, and work inside a product.

Supervised fine-tuning supplies the behavior. You collect pairs of instruction and good answer, then train the model to imitate the answer by gradient descent. This is supervised learning on expert demonstrations. Take a reasonable base model and public instruction data such as OpenAssistant or OpenHermes. Train with solid hyperparameters. You get a chat model that follows instructions and trails the best released ones.

Instruction data arrives from three sources. Each one charges a different price.

• Benchmark data converted to instruction format, the FLAN style, is large and cheap, and it teaches short multiple-choice answers.
• Human-written chat data, the OpenAssistant style, is slow and expensive, and it teaches what a real conversation looks like.
• Model-generated data, the Alpaca style, is cheap and consistent, and it teaches the quirks and hallucinations of the teacher.

Diversity is the price on the Alpaca style. A seed set of human instructions goes into a strong model such as InstructGPT. More instructions come back, and they stay short and similar to each other.

Safety tuning exists because people misuse models for scams, misinformation, and harm. A product also needs the trust of its users and its advertisers. Safety data is instruction data where the correct answer is a refusal or a safe redirection. A few hundred well-built examples mixed into the instruction set change how the model refuses. The effect is larger on a strong base model.

Instruction data no longer waits for a final stage. Engineers mix instruction-like data into late pre-training while the learning rate decays. Stage one stays classic, on web, code, and books. Stage two adds higher-quality instruction data and keeps training, pulling in Wikipedia, QA, chats, code SFT, and StackExchange. A small final SFT pass on the best instruction set is optional.

Mid-training uses instruction data in bulk. Keeping pre-training data in the mix cuts catastrophic forgetting. Each instruction example also returns more value. Many models shipped as base models already saw instruction-like data in mid-training. The clean split between base and chat is gone.

Fine-tuning on answers the model cannot support teaches invention

Quality in instruction data depends on what the model already knows. The same answer is good training for one model and a trap for another.

Take a training example that asks for an introduction on monopsony and ends with a specific citation. That answer teaches two things at once. One is a fact claim, that this reference exists and fits. The other is a shape, that a complex answer ends with a citation-like string.

A model that does not hold the reference learns only the shape. It then invents references that match the shape. The run taught it to copy the form of a good answer.

Post-training data more polished than the knowledge of the model teaches it to fake competence. Instruction tuning and RLHF both reward confident detail. Careless data teaches the model to always answer, even where a refusal is correct.

Good data sometimes says that it does not know. When the fact is missing, the correct target is a short admission or no answer at all. On-policy RL points the same way. It trains on the outputs of the model itself, and the updates land where the model already has some ability.

Post-training changes behavior more than knowledge. Mid-training adds a little, and a small SFT set with an RLHF run does not replace broad pre-training. The data notes put the ceiling in the corpus. Post-training decides how much of that ceiling a user ever sees.

RLHF changes the objective from matching a distribution to scoring well

Pre-training and SFT assume a target distribution of good completions. Next-token prediction pulls the model toward it. Success means the model looks like its training data.

RLHF drops the target distribution. You define a reward over the prompt and the answer. The model becomes a policy that picks answers scoring high on that reward. Success means a high score from a human or from a reward model.

A KL penalty holds the policy near a reference, usually the SFT model. Remove it and the policy drifts.

Two things make judgments worth the switch. A supervised example needs a full expert answer for every prompt. A preference label needs one choice between two candidates, and it costs less to produce. People also write worse answers than the ones they choose when shown options. That distance is the generator-validator gap, and RLHF runs on it.

The InstructGPT-style pipeline is short. Start from an SFT model and sample several answers for each prompt. Show annotators the pairs and ask which answer is better, with a tie or both bad allowed. Fit a reward model to those choices. Run an RL algorithm that raises the score of the policy.

A rater with one minute cannot check facts

The guidelines ask for helpful, truthful, and harmless answers. The rater also judges style, since the answer must stay polite and clear. Toxic content and hallucinations are supposed to lose.

The budget is about one minute for each example. Correctness is the part that does not fit, and math or fact-heavy answers are the worst case.

Checking one factual claim takes four steps.

• Understand the question that was asked.
• Read both answers closely.
• Split each answer into separate claims.
• Check those claims against a reliable source.

Four steps for each claim, times several claims, inside one minute. The rater judges what is visible in the time available. Longer, confident, well-structured answers feel better. A long wrong answer beats a short correct one when nobody checks the facts.

Some raters paste the pair into GPT-4 and copy the verdict. The judge and the model stop being independent, and self-preference rides back into the training data.

The work is often outsourced, and the pay runs low against the cognitive and emotional load. Culture and religion shape what a person calls a good answer. RLHF is the last high-impact step in the pipeline. When most raters share a region or a faith, the outputs drift toward those values.

The open question is whose preferences count. Crowd-worker demographics set part of the answer. Company policy and the bias of an AI judge set the rest.

AI judges agree with humans about as often as humans agree with each other

A strong model reads two answers and picks one, faster and cheaper than a person. Experiments put its agreement with human labels at about the level two humans reach.

Large feedback sets follow from it. UltraFeedback is one, and systems such as Zephyr and Tulu train on judgments from an LLM.

The preferences arrive in two kinds, and pipelines mix them. Off-policy preferences cover outputs from many models. On-policy preferences cover the outputs of the model being trained. The mix decides which part of the answer space the reward model has ever seen.

Both sides of that pipeline prefer longer answers. Humans favor long, list-like answers, and LLM judges favor them too. The preference holds even when the long answer carries more hallucinations. RLHF then pushes the model toward verbosity.

The evaluation note found length bias deciding leaderboards without ever appearing on them. It arrives here through the training door. Track length and style in the training data and in the evaluation. Treat any win that moves with length as unproven.

One score never settles a post-training question. Many benchmarks barely move with answer length. Chat evaluations move with verbosity and vibes, and MMLU-style scores miss style and safety. A working pipeline reads more than one measurement before it ships.

DPO gets most of PPO's gain with a supervised objective

The true reward is never observed. What arrives is a set of pairwise choices, each one saying that this answer beat that one.

Bradley-Terry turns those choices into a score. Each answer gets a number. The chance a person picks one rises with the score difference, through a logistic function.

Fit that model by maximum likelihood on the preference data and you have a reward. Then push the policy toward a higher score. The policy gradient raises the probability of high-reward answers and lowers the probability of low-reward ones.

That loop is unstable on its own. Variance is high, and a policy that trains again on old data drifts. PPO holds it together. An advantage subtracts a baseline, clipped ratios bound each update, and a KL penalty keeps the policy near the reference. It works, and it is a lot of machinery to build and tune on a large model.

DPO removes the loop. Under KL regularization the optimal policy and the reward have a closed-form relation. The reward becomes a log-ratio of the policy against the reference.

Training then runs on chosen and rejected pairs directly. Raise the log probability of the chosen answer and lower the log probability of the rejected one. The explicit reward model and the PPO loop both drop out. The objective looks supervised, and open research uses it heavily because it works.

The preference set is the objective now. Whatever the labels rewarded, the gradient rewards.

The Builder Test

Name the proxy before you trust an improvement. Take the last preference set or instruction set you shipped.

Write two lines about it. The first names what a label rewarded, whether that is length, structure, politeness, or a citation at the end. The second names what no label checked.

Then write the sentence a model can follow to win the first line while failing the second. If you can write that sentence in a minute, a training run finds it faster. Fix the data, or add a check that a program can run. Measure the fix against the behavior you named.

What Carries

Post-training reshapes behavior with a small amount of data. The same step that makes a model useful and polite installs hallucination and bias without a signal. Every choice of data, labeler, and feedback rule shows up in how the model speaks, refuses, and invents.

The model learns what the system rewards. Whoever wrote the rubric meant something else. The reward is what you optimize now, so the next question is what happens when you push on it hard.

A web filter reads every document in the raw pool and keeps a small fraction of them. Each read must be almost free, or the filter spends the compute it was built to save.

That constraint shapes the whole data pipeline. You crawl, you parse HTML into text, you filter for language and quality and toxicity, and you drop the duplicates. Four jobs, one cheap pattern.

The filter must cost orders of magnitude less than the training

Most filters run one recipe. You hold a small target set T that you like, and a huge raw set R that you have, Common Crawl for example. You want the part of R that looks like T. Train a cheap model on T, sometimes on R as well, and score every document in R.

You spend the score in one of two ways. Keep the documents above a threshold, or resample with a probability tied to the score. The filter must also generalize past T, because T is small and the pool it judges is the web.

Now price it. If you keep 1 percent of the web, the filter reads a hundred documents for every one that reaches training. Multiply the per-document cost by that hundred before you compare it against the run. Heavy scorers lose on that arithmetic before anyone argues about their accuracy.

Three cheap scorers, one pattern

An n-gram model estimates the probability of a word from the n-1 words before it. Count the n-grams in a corpus, then read the next-word probability off the counts for that context. Sparsity breaks the naive version. Many reasonable n-grams never appear, the problem grows with n, and raw counts return a zero for text that is fine.

Smoothing fixes that, usually Kneser-Ney, the version KenLM implements. When the counts for a long context are weak, the model falls back to shorter contexts and interpolates across the lengths. Train KenLM on clean text, usually Wikipedia, then score raw pages by probability or normalized perplexity. Low perplexity means the page reads like the training corpus, and high perplexity means junk.

CCNet and the early LLaMA pipelines ran exactly this. They trained KenLM on Wikipedia, scored Common Crawl paragraphs by perplexity, sorted, and kept the best portion. The model catches obvious garbage and non-language text. Repetition fools it, because the window is a few words wide.

fastText is the discriminative option and it is built for speed. A plain bag-of-words classifier over word counts works, and the weight matrix grows enormous. fastText maps each word to a low-dimensional embedding, averages the embeddings into one document vector, and runs a linear classifier on that vector.

Word order comes back through hashed n-grams. Build word n-grams, bigrams for example. Hash them into a fixed number of bins and treat the bins as extra tokens. Different n-grams collide in one bin, and the averaged weights stay usable anyway.

The filter itself is binary. The target set T supplies the positives and random samples from R supply the negatives. Train it, run it over R, and keep the documents that score high on target.

Importance sampling replaces the boundary with a ratio. You want samples from a target distribution p, and you can only draw from q. So draw x from q, weight it by p(x) divided by q(x), and resample by weight. T is your sample from p and R is your sample from q.

T is small, so fitting a rich p is hard. The cheap version uses hashing again. Hash tokens or n-grams into buckets, count the bucket frequencies in T and in R, and smooth both into probabilities. Score a document by multiplying its bucket probabilities under each distribution, then take the ratio.

That keeps more diversity than a sharp in-or-out classifier, because it matches the full shape of T rather than a boundary. The three scorers differ in what they model and agree on the move. Each one learns how much a document looks like T, and the score decides what survives.

Use an expensive model once, then distill the signal

Quality has no single definition. Grammar, coherence, low spam, and educational value all get called quality, and a cheap model approximates any of them at best. So teams define quality by example. The positive set is the definition.

The GPT-3 pipeline used curated sources as positives, books and Wikipedia and WebText, with random Common Crawl as the negatives. Train a linear classifier and filter Common Crawl by its score. Smooth sampling sometimes takes the place of a hard cut.

The first LLaMA pipeline moved the positives one hop out. Pages linked from Wikipedia became the target and Wikipedia itself stayed out of the set, with random crawl pages as negatives. The classifier kept what it labeled positive.

Phi-1 paid a strong model for the definition. Start with the Python subset of The Stack. Ask GPT-4 one question of each file: "How educational is this file for a student learning basic coding concepts?" About 100k labeled files define the target.

GPT-4 over the whole subset is out of the question, so distill it. Compute embeddings with a pre-trained model. Train a random forest on those embeddings to imitate the GPT-4 labels. Run the forest over the full Python subset.

The unfiltered subset reached about 12 percent on HumanEval after 96k steps. The filtered subset reached about 17 percent after 36k steps. The filter bought the score and the steps together.

The pattern holds beyond Python. Pay a strong model once on a small sample to define the signal. Distill the signal into a classifier you can afford. Run the cheap one over everything.

Cheap filters see local patterns only

These scorers read local word patterns. Long-range coherence and factual correctness sit outside what they measure. Shuffle the sentences inside a document and the n-gram statistics barely move, so the shuffled version scores about the same. Text written to beat the filter passes it.

So they belong early, where the job is to remove the worst of the web for almost nothing. The deeper quality signals arrive later, once the pool is small enough to afford them.

Language identification runs the same machinery. fastText ships a pre-trained language identification model trained on multilingual sources, and it returns a probability for each language. Pipelines keep the pages where the probability of English clears a threshold, 0.5 for example. The identification model handles ordinary sentences well and struggles with short snippets, code, formulas, dialect, and code-switching.

The reason to bother is arithmetic. Compute is fixed, so tokens spent on other languages come out of the language you care about. BLOOM trained on about 30 percent English and shows the trade against an English-only model of the same size.

Toxicity filtering is the same shape once more. The Jigsaw Toxic Comments set labels Wikipedia talk-page comments as toxic, severe toxic, obscene, threat, insult, and identity hate. Dolma-style pipelines train one fastText classifier to separate hate from safe text and a second to separate NSFW from safe. Both run over the raw text, and the pipeline drops or down-weights the high scorers. Both miss where the context carries the meaning, and both hold up as bulk filters.

Filtering changes the training distribution. Every threshold is a claim about what the model must learn. The data note made that point about the domain mix, and here it narrows to one number in a scoring pass.

Deduplication is a different job from quality filtering

Quality filtering says this document is bad, remove it. Deduplication says this document is fine, keep fewer copies. Both run in the same pipeline, and the questions they answer do not overlap.

Duplicates waste compute, because the run trains again on text it already read. They also raise memorization. Repetition increases the chance of verbatim recall, and verbatim recall is where the copyright and privacy problems start.

The web supplies both kinds. Exact duplicates are identical text across mirrors and reposts. Near-duplicates are the same page after a small edit: boilerplate, templates, localized versions, licenses, and mass-copied artifacts.

Three choices define a deduplication run before any algorithm shows up.

• The unit can be a sentence, a paragraph, a fixed-length span, or the whole document.
• The match can be exact, high overlap, or semantic similarity.
• The action can remove every copy but one, or cap the frequency.

The hard part is the count. Pairwise similarity over billions of documents is impossible, so the working methods turn comparison into hashing and bucketing.

Exact deduplication is hashing with a Bloom filter

Exact deduplication is three steps. Hash each unit, usually a paragraph or a span. Group by hash. Keep one member of each group.

C4 runs this on three-sentence spans, and a repeated span survives in one place only. That produces strange edits inside otherwise fine documents, and the cost stays near-linear in the number of spans.

Holding every hash in memory gets expensive, so pipelines reach for a Bloom filter. The filter is a bit array of length m with k hash functions, and every bit starts at zero. Insert sets the k hashed positions to 1.

The query is asymmetric.

• If any of the k bits is 0, the item is new, and that answer is never wrong.
• If all k bits are 1, the item is probably a duplicate, and sometimes that answer is wrong.

False positives come from collisions, and the rate depends on m, n, and k. One k is optimal for a given m and n. Dolma runs paragraph-level exact deduplication this way, with the false positive rate tuned to about 10^-15. A false positive drops a paragraph the corpus never saw before, so the rate goes very low. Standard Bloom filters only insert, which fits a one-pass pipeline.

MinHash turns similarity into collision probability

Near-duplicates need a similarity measure and a way to find the similar pairs without comparing every pair. The measure is Jaccard similarity over sets. Turn each document into a set of word shingles or n-grams, then divide the intersection size by the union size. Pipelines call two documents near-duplicates above a high threshold, often about 0.99.

Computing that over all pairs is out, so MinHash converts it into hashing. Pick a random hash over every possible set element. Define h(S) as the element of S with the smallest hash value. The probability that two documents collide is their similarity.

Repeat with independent hashes, and the fraction of matches estimates the similarity. Banding turns that estimate into a lookup. Compute n MinHash values per document as a signature. Split the signature into b bands of r rows, with n equal to b times r. Two documents become candidates when every hash inside one band matches.

One band matches with probability s raised to r, where s is the Jaccard similarity. The chance of becoming a candidate follows from that.

The pair r and b set the shape of the cutoff. A larger r sharpens the transition and raises the effective threshold. A larger b creates more collisions at lower similarity. Buckets come out the other end, and any exact comparison you still want happens inside a bucket.

The evaluation note named decontamination as a duty, and this is the machinery that does it. A benchmark item sitting in the training corpus makes the score on that benchmark meaningless. Run MinHash with the benchmark on one side, and drop the crawl documents that land in the same bucket. The operation is identical. Only the target set changes.

Paraphrases escape all of it. A paraphrase carries the same meaning in different words, so the n-gram sets barely overlap. Embedding the documents and searching that space finds them, at a cost above hashing. Loose thresholds there erase useful diversity, so this stage needs a stricter threshold than the exact ones.

The Builder Test

Price the filter against the run before you write it. Measure the cost of one forward pass. Multiply by the size of the raw pool. Compare that number against the training run it protects. If the two numbers sit within one order of magnitude, the filter has no benefit left to give.

Then read what it does. Sample the documents it kept and the documents it dropped, and read both samples yourself. If the sample surprises you, the model learns that surprise across the whole corpus.

What Carries

Cheap scoring buys expensive compute back. Every gate you can afford to run over the raw pool is compute the training run never spends on text you did not want.

Write the threshold down next to the run. Anyone reading the result later needs to know what the corpus was allowed to contain. The corpus is clean at that point. The next question is what the model does with an instruction after it reads all of it.

Two 1B models trained on the same 100B tokens land several points apart. One draws from raw Common Crawl. The other draws from a filtered pool like DCLM. The filtered model scores higher across knowledge and math benchmarks, on the same compute.

Two models on the same token budget land points apart, which is why labs stay vague

That gap is data quality alone, and it grows with the size of the run. The corpus sets the ceiling. Extraction, filtering, deduplication, and mixture decide where the ceiling sits, and the architecture works underneath it.

Data decisions also cost almost nothing next to the training run. They are the cheapest large effect available to a team.

Model papers describe the architecture in detail and stay vague about the corpus. LLaMA 3 names broad sources and rough filtering, and stops there. Two reasons keep the recipe private.

• The mix and the filters drive quality, so the recipe is the advantage.
• Fewer disclosed specifics leave fewer targets for a lawsuit.

The overview said every stage prices the next one. Data is the stage that prices all of them. Foundation models cut manual labeling and moved that work into selection, cleaning, filtering, and mixing.

Data work also spreads across a large team. Architecture work stays with a small core. When compute and architectures converge, the corpus is what one lab has and another does not.

Common Crawl is a biased snapshot, and the extractor is a quality lever

Common Crawl is a non-profit that started crawling in 2007. It runs a crawl every month, and each one covers billions of pages. Coverage is incomplete and skewed, and the result is a sample of the web.

The crawler starts from a seed set of URLs and keeps a frontier queue. It fetches in parallel, follows links, and adds what it finds. Heuristics decide revisit frequency. Robots.txt holds down the load.

Each crawl ships in two forms. WARC keeps the raw HTTP responses. WET keeps extracted text, and that extraction is lossy. The same crawl also carries harmful content, heavy duplication, and gaps where important sources are missing.

Turning HTML into text is a major quality lever. Extractors disagree about what is content and what is boilerplate. Swapping one extractor for another moves benchmark scores by multiple points.

Teams drop the default WET text for that reason. RefinedWeb ran Trafilatura over Common Crawl and argued that filtered web alone is enough. NeMoTron CC later switched to jusText to keep more usable text.

The evaluation note named this failure. A pipeline change moves a benchmark by multiple points, and no leaderboard can say which stage bought the move. Report the extractor next to the score. Without it the number has no owner.

Every filter encodes a belief about what good looks like

GPT-2 built WebText out of a proxy for worth reading. Take Reddit posts above 3 karma, follow the outbound links, and download those pages. That gave about 8 million pages and about 40 GB of text. The belief inside the rule is that upvotes track quality.

OpenAI kept WebText private. OpenWebText is the open reproduction, and many teams train on that instead.

CCNet deduplicates, runs language identification, then scores each document against Wikipedia statistics and keeps what resembles Wikipedia. Wikipedia carries sourced facts, few recipes, and little first-person advice, and its notability rules keep out niche topics. A filter tuned to it inherits that shape.

C4 trusts grammar. Google built the Colossal Clean Crawled Corpus from one Common Crawl snapshot for T5. It keeps English lines that end in sentence punctuation and drops pages under three sentences. The rules also drop pages with bad words and lines with braces, which removes most code with the spam.

The two blind spots run opposite. CCNet drops useful text that reads differently from Wikipedia. C4 keeps spam that is well formed.

GPT-3 replaced hand rules with a learned filter. Its mix was processed Common Crawl, WebText 2, two books corpora, and Wikipedia. Positives came from WebText, Wikipedia, and books, and negatives came from random Common Crawl pages. The classifier kept the crawl pages that looked like the positives. The whole recipe came to about 400B tokens.

That pattern is now standard. Define what good looks like, train a classifier on it, then mine a better subset from a huge pool. LLaMA trained its classifier to match sources cited by Wikipedia and reached about 1.2T tokens.

Gopher went the other way. DeepMind wrote quality rules by hand and used SafeSearch to cut harmful content. It avoided model-based filters for one reason: a weak filter model stamps its own bias onto the corpus. MassiveText held MassiveWeb, C4, books, news, GitHub, and Wikipedia, about 10 TB in all. Gopher trained on about 300B tokens.

Open reproductions made the filters inspectable. RedPajama v1 rebuilt the LLaMA mix with data and code released, and Cerebras published a deduplicated subset near 627B tokens. RedPajama v2 processed about 30T tokens and attached several quality signals so teams can run filter experiments.

RefinedWeb pairs strong extraction with Gopher-style rules and fuzzy deduplication. It holds about 5T tokens and released about 600B of them publicly. FineWeb runs rule-based filtering, basic anonymization, and deduplication across every Common Crawl dump and stops near 15T tokens. It exists as a base layer for teams that filter harder.

DataComp made filtering itself the measured object. The DCLM pool processes every dump into about 240T tokens. DCLM Baseline cuts that to about 3.8T with rule-based and model-based filters.

The model-based filter is a FastText classifier. Positives are OpenHermes and ELI5, negatives are random FineWeb samples, and the classifier scores the whole pool. DCLM Baseline beat RefinedWeb and other datasets on many benchmarks, and OLMo adopted it. Filtering now runs with a model in the loop. The filter carries the biases of the model that built it.

3.8T tokens is too small for a very large run. NeMoTron CC kept the DCLM idea and went back for tokens that DCLM threw away. A large NeMoTron model rates educational value, and the team distills that rater into a faster classifier. The DCLM classifier supplies a second score.

NeMoTron CC then buckets documents by score and samples across all buckets to hold diversity. Some pipelines also rewrite low-quality documents into cleaner text with an LLM. From the high-quality ones they generate questions, summaries, and key-information pairs.

The result is about 6.3T tokens from Common Crawl, beating DCLM and FineWeb on average benchmarks. A 1T-token high-quality subset performs best of all.

Karma, Wikipedia similarity, sentence punctuation, and an educational-value rating are four claims about good writing. A repeated document teaches memorization. A bad extraction teaches noise. The model learns the claim you encoded.

An open dump is an attack surface

Wikipedia is heavily edited and multilingual, and it publishes downloadable dumps on a schedule. An attacker can time a schedule.

An attacker edits a page before the dump runs. The edit lands in the snapshot. A revert on the live site cleans the page. The dump still carries the poison.

A model trained on that dump learns the association the attacker planted. Nobody auditing the live page can see the text that taught it.

Special sources buy capability and carry licenses

Stack Exchange ships a question, an accepted answer, and votes. A question comes in and an answer comes out, which is the shape of instruction following. The votes are a quality signal that costs nothing to compute. Dumps exist, and commercial use can need a license.

GitHub is the code source, and most of it needs sorting. Many repositories are tiny, abandoned, forked, or duplicated. Three decisions build the corpus: what counts as code, which licenses pass, and how forks get deduplicated.

The Stack read repository names from GitHub Archive and cloned about 137M repositories. It kept the permissively licensed ones, deduplicated them, and produced about 3.1 TB of code. Every source runs that path: live service, snapshot, processed training set.

Papers add dense scientific text. PubMed Central publishes open-access biomedical work, and Semantic Scholar aggregates across many sources.

Books buy long context. Project Gutenberg holds tens of thousands of public-domain books, and PG-19 comes out of them. Books3 came from a shadow library and carried copyrighted works. It was removed after complaints.

LibGen and Sci-Hub host large collections outside copyright, and lawsuits claim that some models trained on them. That is the second reason for the vagueness, and it is the one with a court date.

The Pile mixed OpenWebText, Stack Exchange, ArXiv, PubMed Central, GitHub, Gutenberg, and Enron emails. It came out larger than the GPT-3 training data. A curated source is easier to control and reuse than a crawl, and it leaves a fingerprint. The Enron mail shows up in how those models write email.

Dolma stacked processed Common Crawl, The Stack, C4, Semantic Scholar papers, Gutenberg, and Wikipedia into about 3T tokens. It ran English language identification, quality filters that started without classifiers, a classifier for harmful content, and deduplication.

Dolma also pulled Reddit submissions and comments through a project that later lost its public API. A source can close after the corpus depends on it.

Copyright and platform terms set the real boundary

Copyright attaches the moment a person posts text. It needs no notice and no registration. It covers expression and leaves ideas free, and the terms run for decades. Most of the web is protected by default.

Registration matters when someone sues. Using protected text needs a license or a fair-use argument that holds. Platform terms bind on their own. A work under an open license can still sit behind a rule against bulk downloading.

Creative Commons keeps the copyright and grants reuse under conditions, which is how Wikipedia is licensed. Direct deals between builders and content owners cover more. Most teams end up mixing licensed data, open data, and a fair-use claim.

Courts weigh four factors.

• Purpose and character, which favors a use that changes the work, and research over commerce.
• Nature of the work, which protects creative writing more than factual writing.
• Amount used, and training copies whole works.
• Effect on the market, which asks whether the new use replaces the original.

A court found Google Books scanning and snippet display fair. The scans served a new purpose, and the snippets did not replace book sales. Builders argue training is the same kind of use, since a model learns patterns and not the text.

Training is the harder case. It copies whole works, and models can memorize and reproduce what they copied. Courts are still deciding.

Price the legal exposure with the compute, before the run starts. A corpus you cannot defend is a run you cannot ship.

Mid- and post-training buy targeted skills for a small share of the tokens

Pre-training takes the huge noisy web corpus. Mid-training takes smaller, cleaner slices aimed at math, code, reasoning, or long context. Post-training does instruction and chat tuning, often with reinforcement learning for helpfulness and safety.

A base model comes out of pre- and mid-training. Post-training produces the chat model that obeys instructions and holds a conversation. The boundaries blur, some pre-training data already reads like instructions, and some pipelines run more than three stages.

Mid-training is a small fraction of the total tokens. AI2 built theirs from filtered slices and instruction sets in the FLAN style. Synthetic data covers math, reasoning, and long context.

The efficiency note priced attention against sequence length, and that price is why long context arrives late. Teams train mostly on short sequences and add long documents at the end. Books, long articles, and synthetic long sequences fill that stage.

Older NLP kept a separate format for every task, and instruction tuning collapsed them into one. SuperNaturalInstructions reformatted more than 1,600 tasks into instruction prompts and outputs. FLAN did the same across many NLP tasks. Both lift benchmark numbers. The prompts stay templated, and user language is messier.

Self-Instruct and Alpaca start from a seed set and let a strong model write both the instructions and the answers. Vicuna trained on ShareGPT logs, which showed that real conversation lifts chat quality. Self-chat and evol-instruct raise the difficulty step by step. Another pattern pulls question-and-answer pairs out of web text with an LLM.

OpenHermes is a curated mix of instruction and chat sets. It also serves as the positive class for quality classifiers, which runs the loop back to filtering.

LLaMA 2 Chat leaned on human-annotated instruction and chat data, and it beat large synthetic sets. Human data is slow and expensive, so teams stretch it with RLHF. Teams also have to keep annotators from using GPT-4 in secret.

LLaMA NeMo post-training mixes public chat sets like WildChat with synthetic conversations from open models. It adds reasoning traces from models like R1 to teach chain-of-thought output. The rest of a post-training set is licensed chat transcripts and data shaped for safety.

The Builder Test

Sample a hundred documents your filter kept and a hundred it dropped. Read them before you trust the mixture.

If the kept pile surprises you, the corpus teaches that surprise to the model. If the dropped pile holds text you wanted, the filter carries a belief you did not choose.

What Carries

The model cannot become better than the signal you give it. Every filter is a claim about what that signal is.

Size alone is a bad measure of a corpus. More tokens make a worse model when they are duplicated, poisoned, low quality, legally risky, or weighted badly. The field still runs on heuristics, and a better pipeline still buys a large gain.

Filtering decides what stays. Deduplication decides how often what stays repeats, and it is a separate job. The mechanics of doing both cheaply are the next note.

A benchmark number carries no meaning apart from the choices that produced it. Two of those choices decide leaderboards without ever appearing on them. Contamination puts the test set inside the training data, so a high score measures memory. Length bias makes an automatic judge prefer the longer answer, even when the longer answer hallucinates more.

Both survive because the reported number hides the pipeline that made it. Rebuild the pipeline and the number becomes readable, or it stops being worth reading.

What you measure becomes the target

Labs are ranked on a handful of numbers. Teams then change the data mixture, the architecture, and the training run to move those numbers. The number stops describing the model and starts designing it.

Four audiences read the same score for four different reasons.

• A company picking a model for one workflow wants to know whether it fits that workflow.
• A researcher tracking progress wants to know whether a training change improved reasoning.
• A policymaker weighing benefits against risks wants to know whether the model is safe enough to deploy.
• A developer mid-run wants feedback fast enough to steer the next checkpoint.

One evaluation cannot answer all four. An evaluation built to satisfy everyone satisfies no one. Pick the question first, then build the setup that fits it. An accuracy number with no context misleads all four readers at once.

Every evaluation is a pipeline of four choices

Four stages sit between a question and a number, and each one adds an assumption that the number hides.

The inputs set the ceiling. Prompt source and coverage live here, along with whether the rare and hard cases appear at all. In multi-turn chat the later inputs depend on earlier outputs, so you cannot script the sequence in advance.

The call decides what you are testing. Zero-shot, few-shot, chain-of-thought, tool use, and retrieval each expose a different part of the same weights. This stage also fixes the unit of measurement. You test a bare language model, an agent with tools, or a full system with scaffolding and routing.

The scoring decides what counts as right. Exact match, F1, pass@k, BLEU, and ROUGE reward different behavior, and each assumes the reference answers are clean. Cost belongs in this stage. Two models with the same score can differ widely in price and latency, and deployment pays that difference.

The reading decides what the number licenses. A score of 91 percent is not a deployment decision until you know which items fail. Separate real generalization from overlap with the training data. Then name what you measured: a model, a system, or a change in method.

Adapting prompts to one model buys realism and costs comparability. A fixed script pushes a chat model into states no user creates. Letting the model drive the conversation looks more like real use. In red-teaming, adapting each prompt to the last reply finds rare failures faster. Both moves make the inputs model-specific, so cross-model comparison stops being fair.

Every stage prices the next one. Evaluation is the stage that prices your reading of all the others.

Perplexity still guides pretraining and misses what users want

A language model assigns probabilities to token sequences. Perplexity measures how well it predicts tokens on held-out text, and lower is better. Pretraining optimizes this objective directly. Perplexity is also the metric the scaling notes fit, so those curves ride on this one number.

Earlier work compared n-gram models against neural models on Penn Treebank and WikiText. Large neural models cut perplexity by a wide margin, and that drop drove scaling.

GPT-2 trained on WebText and scored well out of distribution on those older benchmarks. That is transfer. GPT-3 and the models after it still report perplexity, and attention moved to downstream tasks.

Two mechanical properties keep perplexity in the loop. Token probabilities give smooth, fine-grained feedback that a discrete accuracy score cannot give. The same machinery reaches downstream tasks: condition on the prompt and score the candidate answers.

Perplexity is a blunt instrument for capability. A small change in behavior often leaves the number flat, so it judges small model changes poorly. A perplexity leaderboard also asks you to trust reported probabilities, and invalid or manipulated probabilities produce improvements that never happened. Name the proxy before you trust the number.

The open argument is whether matching the true data distribution is enough on its own. One side treats it as the whole problem. The other side calls that route inefficient and badly aligned with the tasks people run.

Multiple-choice benchmarks that compare probabilities over completions are perplexity under another name. LAMBADA asks for a missing word that needs long-range context. HellaSwag asks for the most plausible continuation as a proxy for common sense. Many draw on web-adjacent sources like WikiHow, so their items sit inside the pretraining corpus already.

Exams saturate, leak, and encode their own bias

MMLU covers 57 subjects with multiple-choice questions pulled from the web. It weighs knowledge heavily, and reasoning without the right facts does not score. Early evaluations used few-shot prompting, before instruction tuning existed. Prompt wording and example order both moved the scores. Instruction-tuned models now score well zero-shot, and parts of the benchmark are saturated.

MMLU-Pro raises the difficulty. It drops noisy questions and moves from four choices to ten. Strong models lose accuracy on it, and chain-of-thought helps on some items and not on others.

GPQA holds PhD-level technical questions that experts write and review. Experts score about 60 percent. Non-experts working with a web search score about 30 percent.

Google-proof means a plain search does not close that gap. Earlier models scored in the high 30s, and newer models score higher. A model can still reach the answer through web-like knowledge patterns.

Humanity's Last Exam is multimodal and hard by design, with prizes for items that defeat models. The people writing those items know how models behave, so the questions drift toward ultra-specific traps. That is a stress test at the far end. No user has a Tuesday that looks like it.

Every exam encodes the taste of the people who wrote it. Saturation, prompt sensitivity, and item bias all move a score that reads as pure capability.

Judges reward length

Open-ended instructions have many acceptable answers and no single reference output. Every automatic score for them is a trade-off written as a number.

Chatbot Arena shows a user two hidden replies and asks which one is better. Pairwise votes turn into ELO-style rankings. The prompts are live, no fixed test set exists to leak, and the human judgment is easy to give. The same design carries user bias, vote manipulation, and the leaderboard distortions that follow.

IFEval checks narrow constraints, like a word limit or a banned token. Those constraints are mechanical to check, so results reproduce across runs. They measure surface compliance, and a nonsense answer that obeys the word limit passes.

AlpacaEval hands the comparison to GPT-4 and reports win rates. Early versions preferred longer answers, and length-corrected variants cut that bias. Its scores correlate with Chatbot Arena. WildBench judges real human-bot conversations with LLMs working from checklists.

Length bias is the clearest case of a judge scoring the wrong variable. It lived inside a widely used benchmark until someone measured it and corrected for it. Assume your next judge carries a bias of the same kind, and find it before you publish.

Agent benchmarks score the model with its scaffolding: multiple calls, planning, tools, code execution, and debugging. The best systems still solve a small fraction of these tasks, and the fraction is growing. Long-horizon planning, tool use, and debugging expose weaknesses that short answers hide. Run one long task before you trust a short-answer score.

A model that refuses everything scores perfectly on safety

Safety benchmarks measure refusal, so a model that refuses everything maximizes the metric and serves nobody. Read safety numbers next to capability numbers, or do not read either.

HarmBench defines harmful behaviors and checks whether models refuse them. AIR-Bench maps safety categories onto laws and policies and probes models against those categories. Both aim at the same target: helpful work done safely, dangerous requests refused.

Two quantities hide under the word safety. Capability asks whether the model can produce harmful content at all. Propensity asks whether it does so under allowed prompts. Closed APIs lower propensity through safety tuning, and open weights expose capability. A user can strip the safety layer or fine-tune away its effect.

Jailbreak results are evidence about brittleness. Optimized gibberish suffixes push models past their safety rules, which puts refusal behavior on a thin layer. Systematic prompting surfaces harmful behavior behind nominal refusals.

Safety institutes run voluntary pre-deployment tests with labs on unreleased models, mostly reusing existing benchmarks. One question stays open: what counts as safe enough. Answer it in writing for your own product before a benchmark answers it for you.

Contamination and label noise break a benchmark without changing how it looks

Models train on corpora the size of the web, and benchmarks live on the web. Overlap is the default state, and the resulting score looks exactly like a clean one.

A filter on shared long n-grams catches the obvious cases. It misses translations, paraphrases, and variants, including the same math problem written in another language. Decontamination and deduplication are one operation seen from two ends of the pipeline. The data notes that follow run that filter across the training corpus. Run the same filter against your test sets and you have the decontamination check.

Many papers report no decontamination check at all. Publish the check beside the score, or state that you skipped it.

Label noise runs the other way. Benchmarks carry wrong answers and broken questions, and correcting those errors often lifts the top scores. Part of the difficulty a leaderboard reports is its own noise.

One more gap separates the tests from the work. Real traffic holds chit-chat, messy questions, spam, adversarial requests, and practical tasks. Benchmarks mostly quiz, where the writer knows the answer and uses it to grade. Users mostly ask, because they do not know the answer and need help. Most of the value sits in asking, and most of the measurement sits in quizzing.

Two projects show what closing that gap costs. Anthropic clustered real conversations and found coding to be a major use case. MedHELM asked clinicians which tasks matter in practice, then built benchmarks for those tasks. Both routes need real user or patient data, and that data carries privacy limits on collection and sharing.

The Builder Test

Write down what the evaluation cannot see, then decide how far to trust it. One paragraph covers it: which prompts it never contains, and which failure mode it does not score. Name the part of the score that the scaffolding earned.

Then attach that paragraph to the score every time you report it. A number that travels without its blind spot is a claim nobody can audit, including you.

What Carries

An evaluation earns trust when you can explain the choices that produced the number. That explanation is the contract, and the number is worth no more than the contract behind it.

Every benchmark score rests on what the model read before it met the test. The next note opens the diet.

Chinchilla put twenty tokens on each parameter, and every team that measured the ratio again landed higher. Meta measured about 39 for LLaMA 3. MiniCPM measured about 192. The Hunyuan analysis measured about 96, counting active parameters rather than total ones.

Twenty tokens for each parameter is a floor

Chinchilla's ideas survive contact with production. Its constants do not. Architecture, data quality, and optimization decide where the ratio lands, and all three changed after Chinchilla.

Every one of those numbers came out of an isoFLOP sweep, the method the scaling-law note laid out. Fix the compute budget, vary parameters against tokens, fit the curve, and take the minimum. The constant that falls out belongs to that setup and to no other.

MiniCPM trained models from 1.2B to 2.4B parameters on far more compute than that size usually gets. The result is a small model that is strong for its size. Hunyuan is a mixture of experts, so its sweep counts active parameters. The method holds for that architecture and the constant it returns is specific to it.

The ratio climbed because of the bill the inference note priced. Serving cost tracks parameter count, and it repeats on every request the model answers. Training happens once. A smaller model trained on more tokens moves spend off the recurring side.

After Chinchilla and ChatGPT, most large labs stopped publishing their scaling recipes. Cerebras-GPT, MiniCPM, and DeepSeek kept publishing theirs. Their write-ups are where the full recipe still appears, so the working practice comes from them.

muP keeps the best learning rate fixed as the model widens

Scaling asks for three things. The isoFLOP sweep buys two of them, the parameter and token split and a predictable loss. muP buys the third, which is hyperparameters that survive a change of size. It holds the best learning rate steady as the width grows, and it does nothing else.

Cerebras-GPT trained from 0.1B to 13B parameters on Chinchilla-style compute-optimal budgets, and put standard parameterization next to muP. On a log-log plot of test loss against compute, standard parameterization wanders around the predicted line. muP tracks the line smoothly.

The learning rate is the cause. Under standard parameterization the best learning rate moves as the model widens. One global value then runs too large at the top end. Tune at a small width and the same rate breaks the wide model.

muP holds two things steady as the width grows. At initialization each activation coordinate stays order-one, so a layer's L2 norm grows like the square root of width. After one optimizer step, the change in each activation coordinate stays order-one as well. Those two conditions define width-invariant training.

Two rules follow. Initialize the weights at variance about one over fan-in. That is Xavier and Kaiming practice, and most code already does it.

Scale each layer's AdamW learning rate the same way. When fan-in is the model width, the layer's rate falls like one over width.

Standard parameterization already initializes close to what muP needs. The learning rate is where the two part.

Embeddings get a fixed scale of their own, and attention carries one more subtlety. Standard attention divides the logits by the square root of d. Some muP treatments divide by d instead, for tighter update stability.

MiniCPM ran the test. They swept learning rates at several model sizes and found a wide good region. The minimum sat in nearly the same place at every size. Their best settings matched Cerebras to about a constant factor, with the residual blocks scaled by depth.

That is why the search moves down in size. A large hyperparameter search runs on tiny models, and the wide model inherits the answer.

MiniCPM searched the aspect ratio, the learning rate, and the core settings on 9M and 30M models. Then it froze those choices and widened. The aspect ratio holds on the way up, so d_model, d_mlp, and head count keep their proportions.

muP fails in named places

Three changes break the learning-rate transfer.

• Extra learnable gains in the wrong places break the width invariance.
• Sign-based optimizers such as Lion do not match the assumptions muP is derived from.
• Strong weight decay can break it too.

Two changes leave it alone. Swapping the MLP nonlinearity keeps the transfer, and changing the batch size keeps it too. Small differences in initialization matter less than the learning-rate scaling.

Large studies tune the learning rate at a small width and then train wider models at that same rate. Under muP the loss-against-learning-rate curves agree across all the widths. Under standard parameterization the borrowed rate runs too hot as the width climbs.

Check the optimizer and the weight decay before you trust a transferred rate. One run reached about 10B parameters and trained successfully. Its learning rate came from small-scale muP sweeps, which is the entire return on the method.

WSD makes scaling experiments cheap

Chinchilla-style analysis needs many runs, several model sizes crossed with several token budgets each. Cosine schedules make that grid expensive.

The learning-rate curve depends on the total number of steps. A checkpoint from a long run is not the end of a shorter one. Every point on the grid pays for its own run from scratch.

WSD splits the schedule in three.

• Warmup raises the learning rate to its full value.
• A long stable plateau holds it there.
• A fast decay drops it to a small value at the end.

The plateau does the work. Rewind to an earlier plateau checkpoint and run the same decay from there. The result is a finished schedule for a smaller token budget. One long run yields a row of shorter ones, and you retrain nothing.

MiniCPM ran its isoFLOP analysis on those rewinds, with the lower-envelope method and a two-variable fit. That is where its ratio came from. DeepSeek runs a WSD-like schedule for the same reason, which is reuse across data budgets.

DeepSeek skipped muP and fit the hyperparameters directly

DeepSeek does not use muP. They scale the learning rate and the batch size against compute, and they fit those laws themselves.

The loop starts small. Sweep learning rate and batch size over a grid at small model sizes, then record the minimum. The minimum comes out broad rather than sharp at each level. Repeat at higher compute budgets and keep the best setting from each one.

Then fit the two trends and read them at the target run. Batch size fits cleanly against compute. Learning rate comes out noisier, and it still fits well enough to use.

Their isoFLOP curves came out smooth. At each compute budget they fit a quadratic in tokens and took the optimum. The trained models matched the prediction, which buys fewer surprises at the size where a surprise costs the most.

muP derives the scaling. DeepSeek measures it. Both spend small-model compute to set the dials on the large run, and you pay one bill or the other.

Critical batch size grows as the target loss falls

MiniCPM also measured where a larger batch stops paying. Past that point, more batch buys no further drop in loss. For each model size they trained across a range of batch sizes and recorded the final loss.

The point moves with the target. Push toward a lower loss and the critical batch size grows, on a relation close to a power law. Name the target loss and the batch size follows from the curve.

The Builder Test

Pick one hyperparameter you did not measure yourself, such as the learning rate in your plan. Find the run that produced it. Name the architecture, the data, and the optimizer that run used.

Then say whether your setup matches that one. The common trap is to carry a curve past the setup that produced it. Data quality, optimizer changes, architecture changes, and evaluation leakage all move the constants.

If the answer is no, the number is the start of a sweep. The plan needs the sweep before it needs the number.

I treat a borrowed constant as a dated measurement and check what changed since the date. The check that settles it is one isoFLOP sweep on your architecture and your data.

What Carries

Borrowed constants expire. Each one was measured under a setup, and your setup differs.

The isoFLOP loop replaces them with a number you own. muP and WSD exist to make that loop cheap enough to run again when the architecture or the data moves.

The run is planned and scheduled. The next question is how you will know whether the result is any good.

Generation is memory-bound. Every sequence you serve drags its own KV cache. Moving that cache is most of what the GPU does between tokens.

Training is expensive and rare. Inference runs continuously. At the volumes a company serves, a small gain per generated token beats a large gain on the training bill.

Chat is the visible use. Evaluation, code completion, batch processing, test-time reasoning, and RL sampling all run the same loop. The serving path carries research work as well as product traffic.

The scaling note priced the training run and left serving out of the budget. Serving is the bill that keeps arriving. Three levers move it: shrink the KV cache, share it, or spend compute to skip decode steps. Everything after that is scheduling.

Three numbers describe a serving system, and they do not move together.

• TTFT is the time from prompt arrival to the first output token. It is mostly prefill, and users notice it first.
• Per-token latency is the gap between output tokens after the first. Users feel it as smoothness.
• Throughput is total tokens per second across all requests. Batch jobs and high-volume traffic optimize this one.

Decode is memory-bound and prefill is not

Arithmetic intensity says which resource runs out first.

High intensity puts the ALUs at the bottleneck. Low intensity puts memory bandwidth there. On an H100 the crossover sits near 295 FLOPs per byte.

Take one matmul. X is B × D and W is D × F. The work costs about 2 × B × D × F FLOPs. In a language model D and F run much larger than B, so the intensity comes out at about B. At B = 1 the matmul becomes a matrix-vector product. Intensity falls to about 1 FLOP per byte, against a crossover of 295.

The note that priced a training run counted FLOPs against a budget. Serving counts bytes moved per token. The bandwidth note showed compute growing faster than memory bandwidth. Decode is where you pay that gap.

In training you hold the full sequence. You parallelize across sequence length and run matmuls large enough to saturate the GPU. Decode cannot borrow that trick, because token t+1 waits on token t. The GPU stays underused unless you raise the batch size or run many sequences.

Prefill and decode split on this axis. Prefill runs the whole prompt at once, parallel over sequence length, with high intensity and compute to spare. Decode adds one token at a time, conditions on the full history, and reads far more than it computes. Prefill is the efficient half. Decode is the bottleneck a production server meets.

The KV cache turns quadratic work into linear traffic

Without a cache, every new token re-attends over all previous tokens. Generating T tokens costs O(T²) work for one sequence. That is too slow to serve.

The cache holds the keys and values of every past token, for every layer and every head. A new token computes its own Q, K, and V, then attends over the cached past. Cost per token drops to O(T).

You pay for that saving in traffic. At every step attention reads large cached tensors out of HBM and does little math on them. That ratio is what holds attention on the memory-bound side.

The cache is also the serving bill. Longer prompts, more users, and larger batches compete for the same memory. The weights are the fixed part of that budget.

A larger batch helps the MLP and does nothing for attention

The MLP during generation has intensity of about B. Raise the batch and the MLP walks toward the compute-limited side, where a GPU earns its price.

Attention does not move. The sequences share one set of weights, and each one carries its own KV cache. KV traffic grows with B, so B cancels and the intensity stays near 1 at every batch size.

LLaMA 2 13B on an H100 prices the dial.

• At B = 1, per-token latency is about 8 ms and throughput passes 100 tokens per second.
• At B = 16, the KV cache grows 16 times. Per-step latency rises, and throughput rises with it.
• Past that point, latency keeps climbing while the throughput returns shrink.
• At B = 256, the cache can pass 80 GB, which is more than the GPU holds.

A small batch buys low latency and low throughput. A large batch buys high throughput, higher latency, and memory pressure.

Scaling out is the easy half. Run M copies on M GPUs, and per-request latency stays about the same while total throughput scales with M. Decode needs no cross-GPU synchronization per token, which is what keeps the copies independent.

Shrink the cache or share it

KV movement dominates decode, and four architecture choices attack that traffic directly. GQA arrived in the architecture note and MLA in the mixture-of-experts note. Both were chosen against this bill.

• GQA cuts the KV heads from N to K and lets several query heads share one. The cache shrinks by about N/K, which is why LLaMA 3 ships it. Push K too low and accuracy drops.
• MLA projects K and V into a latent space before caching them. DeepSeek V2 compresses a 16,000-dimension KV space to 512.
• CLA shares the KV projections across several layers, so those layers share one cache and one stream of traffic.
• Local attention keeps a sliding window of size K, so the cache grows with the window instead of the sequence. A pure local stack loses long-range dependencies. Designs answer that with mostly local layers and a full-attention layer every sixth layer.

Another line of work fixes the state size instead of trimming it. State-space models carry a fixed-size state and update it for each token. Each step costs linear time and constant memory. Early versions were weak at associative recall, and Mamba works well on language at modest scale.

Linear attention approximates softmax with kernel features and keeps a running state, again constant memory per step. Some large systems run mostly linear and local attention, with a few full-attention layers for quality.

Quantization is the other lever on bytes. The ladder runs FP32, BF16, FP8, INT8, INT4. Fewer bits per weight moves fewer bytes, and it helps the weights and the cache fit at all. Too much quantization costs accuracy, and some weights carry outliers that need higher precision.

AWQ calibrates to find the weights that matter, drops the rest to INT4, and reaches about a 3× speedup. A 15B model pruned to 8B, then distilled from the large model, keeps benchmark quality close. Both trade fidelity for speed, and both work while the quality loss stays small.

Speculative decoding buys about two times and changes nothing about the output

Checking tokens in parallel costs less than generating them one at a time. Speculative decoding spends that difference.

The target model Q is large, accurate, and expensive. The draft model P is small, cheap, and similar enough in behavior to guess what Q says next.

• P proposes K tokens autoregressively from the current context.
• Q scores all K tokens in one parallel pass.
• Accept each proposed token with the ratio Q(token)/P(token), clipped at 1.
• On a rejection, sample the corrected token from Q.

The output is an exact sample from Q. The distribution matches a run of Q alone, so quality is not part of the trade. When P is fast and stays close to Q, acceptance runs high. Speedups of about 2× or more are common.

Medusa drafts several branches at once. EAGLE guides the draft with signals from the target.

Real traffic is a scheduling problem

Production traffic is irregular. Requests arrive at unpredictable times, and prompts and generations differ in length. Many requests share a prefix.

Dynamic batching runs one step for every active sequence. The scheduler drops finished requests and admits new ones, so the batch is rebuilt continuously.

Selective batching splits the step. Attention needs the KV history of each sequence, so mixed lengths stay awkward there. MLP work does not care, so the server flattens token vectors across sequences and runs shared matmuls.

Sequence lengths are unknown when a request arrives. One contiguous buffer per request wastes memory while the request is short, and leaves a hole when it finishes.

PagedAttention treats the KV cache the way an operating system treats memory. The cache splits into fixed-size pages, and each sequence holds a list of pages instead of one block. The server allocates pages wherever space is free and reclaims them when a sequence ends.

Shared prefixes get the same treatment. The server stores the shared pages once and counts references to them. When two sequences diverge, copy-on-write allocates new pages for the divergent part alone.

High throughput does not imply low latency. A server can push many tokens per second while some requests stall in the queue. Batch policy and scheduler decide who waits.

The Builder Test

Split every serving metric into prefill and decode before you change anything.

• Report TTFT and per-token latency separately. One blended latency number hides which half is slow.
• Measure the KV cache at your real batch size and context length. If it does not fit, the batch size is already decided for you.
• Name the side each optimization touches. A kernel that speeds up prefill does nothing for a decode-bound server.

If you cannot name the slow side, you are tuning by guess.

What Carries

Inference speed comes down to what you can reuse and what you can fit. The KV cache is both. Design as though memory is the constraint, and cut the bytes you move per useful FLOP.

Serving cost now sits in the run plan next to the training bill. The next note reads the measurements production teams published once they planned that way.

Test loss falls on a straight line on log-log axes. That line is predictive enough to pick model size and token count before you spend the budget.

A frontier run is expensive, so you make the choices before it starts. Train many small models and fit the curve. The fit picks architecture, hyperparameters, data mixture, and the split between parameters and tokens. Cheap measurement buys the expensive decision.

Loss falls on a straight line on log-log axes

The line has a formula with three fitted numbers: a floor, a coefficient, and an exponent.

Here n is the dataset size and alpha is positive. L infinity is the error that more data does not remove. Drop that term from the fit and the extrapolation promises loss the large run never reaches. The same shape holds when you plot against parameters or against total compute.

Across many domains and models the loss stays monotone and log-log linear over wide ranges. Inside those ranges the measurements show three regions. Tiny datasets sit near random, a wide middle band follows the power law, and the curve flattens at the floor. Fit inside the middle band, because the two ends bend any fit that includes them. Hestness and colleagues reported that shape in 2017 for translation, speech, and vision.

Fitting a curve and extrapolating it predates this field by decades. A Bell Labs line of work in 1993 argued that full training on huge datasets costs too much. It told you to train smaller systems, fit an irreducible error plus a decaying term, and extrapolate. Banko and Brill showed smooth gains from more data in early NLP. That picture still frames how people plan runs today.

Classical learning theory answered the same question with worst-case bounds from VC dimension and Rademacher complexity. Those bounds run loose. Modern practice measures the loss curves of real networks and fits the formula directly.

The slope reads as difficulty

Strip the floor and the reducible part carries the slope:

On log-log axes that is a straight line with slope minus alpha. For a baseline, estimate the mean of a Gaussian with variance sigma squared from n samples. The squared error falls like one over n, which is a slope of minus 1.

The reported slopes for deep networks are far shallower.

• About minus 0.095 for language modeling
• About minus 0.13 for machine translation
• About minus 0.3 for speech

Each one is slower than minus 1 and slower than minus 0.5. Neural networks behave like non-parametric estimators in a high-dimensional space. An estimator that bins a D-dimensional region and averages locally gives rates near n to the minus one over D. A large D flattens the slope.

Read the slope as difficulty. A shallower slope means a higher effective dimension, and language modeling is the shallowest of the three. A unit of loss in language modeling costs more data than the same unit in speech.

A better data mixture moves the offset and leaves the slope alone

Change the data mixture and the whole curve moves up or down. The slope moves only a little. The offset tells you how good the mixture is at every size.

So compare mixtures with small runs and take the better offset. The advantage continues on the runs you cannot afford to repeat.

When fresh data runs out, repeat what you have. Repetition helps, and each epoch returns less than the one before it. After a few epochs the gains drop sharply.

Price the repeats by replacing the raw token count with an effective sample size. That effective size grows more slowly as the repetition grows. The second pass over a corpus is worth less than the first.

At a trillion tokens the choice gets concrete. You repeat the clean sources, or you add noisier new ones. Each source mixture carries its own curve, so fit both and let the curves set the repetition and the growth.

Small runs decide architecture, optimizer, and shape

Kaplan-style comparisons train several architectures across compute levels. The transformer line sits below the LSTM line by an almost constant factor. For the same loss, an LSTM pays much more compute.

Efficiency decides how much scale you can afford. The tokenizer note made that case for the input, and this curve makes it for the architecture.

Google ran architecture sweeps that compared many variants against a transformer baseline. A few variants beat it, including GLU and mixture-of-experts. Most of the rest added overhead and left the scaling where it was.

Optimizers separate the same way. Across sizes the Adam curve sits below the SGD curve, and the gap holds as an almost constant factor.

Very shallow transformers do poorly. Past a small depth threshold, a broad band of depth-to-width ratios performs the same, with matching slopes and small offsets. Tune the aspect ratio on small models and trust it on the large one.

Batch size and learning rate carry their own curves, and the second scaling note measures both at production sizes.

Chinchilla lands at about twenty tokens for each parameter

A fixed compute budget forces a split between parameters and tokens. A tiny model cannot use a huge corpus. Reverse it and you waste the capacity, because the model stays undertrained.

The joint fit gives each axis its own decaying term:

Here n counts tokens and N counts parameters. These surfaces fit real measurements and predict a large run from small ones.

The compute side comes from the accounting note. A training run costs about six times parameters times tokens in FLOPs, written here with D for the token count.

Both optima grow near the square root of compute, so the two stay proportional to each other. Double the budget and you buy a bigger model and more tokens in the same breath.

Chinchilla measured that split for dense language models with three methods.

• Lower envelope: plot loss against compute for many runs. Read N and n at the best point of each budget, then fit how both scale with FLOPs.
• IsoFLOP sweeps: for each budget, sweep the model size and adjust the tokens to hold the FLOPs fixed. The loss against N is U-shaped, so fit its minimum.
• Surface fitting: fit the joint surface and solve for the compute-optimal point.

The first two agreed. The third carried a bug in the fit from the original paper. Correct that fit and all three agree. The rule stands: for compute-optimal dense training, use about twenty tokens for each parameter.

Training-optimal and product-optimal are different budgets

Chinchilla minimizes the training loss for a fixed training compute budget. That answer is training-optimal. You pay training compute once. You pay inference on every request.

Products pay for inference, and that cost grows with the parameter count and with the context length. When inference dominates, you buy a smaller model trained on more tokens for each parameter. You accept the higher training cost.

The published ratios bracket the choice. GPT-3 sits near two tokens for each parameter, undertrained by the Chinchilla rule. Chinchilla sits near twenty. Newer systems train past twenty to cut the serving cost for each unit of quality.

The Builder Test

Name the decision the curve changes. If it moves no model size, no token count, no compute budget, and no risk, the plot is decoration.

Control the runs behind the curve, with one axis moving and the rest held fixed.

The real choice stays the same at every budget. You are spending fixed compute across parameters, tokens, and quality. The curve makes that split arguable in advance.

What Carries

A scaling law is a planning tool. It holds while the measured assumptions hold, and those assumptions are the part you must write down.

The constants move when data quality moves, when the optimizer changes, when the architecture changes, or when the evaluation leaks. Extrapolating past the setup that produced the curve is where the plan breaks.

The fit is cleanest on cross-entropy and held-out perplexity from the training distribution. Downstream benchmarks behave less smoothly. A model that scales well in perplexity can still be weak on capabilities.

Scaling describes training-like regimes well. Far outside that regime the behavior turns irregular, and on adversarial or out-of-distribution tasks it can reverse. The Inverse Scaling Prize collected tasks where larger models do worse. The task often punishes a behavior that scaling strengthens, such as copying.

The rule optimizes training compute and stays silent on what serving costs. The KV cache note prices that bill.

Collectives set the step time. An all_reduce across four ranks measured about 277 GB/s, against an H100 NVLink peak near 900. That gap is normal. The communication in a parallel plan is a number you measure, not a number you assume.

Keep the work near the data

A node usually holds 8 GPUs. Inside each GPU the streaming multiprocessors do the math, and a ladder of memory feeds them. Every rung down is larger, slower, and further away.

• L1 and shared memory: very small, very fast, one per SM.
• HBM: larger and slower than L1, one per GPU.
• NVLink: fast, and it connects the GPUs inside one node.
• NVSwitch: slower, and it connects nodes to each other.
• PCIe and Ethernet: older, slower, and heavier on overhead.

In the best case the data already sits in L1. Otherwise it comes from HBM. On a multi-GPU node it can come from another GPU, and that rung is where a fast kernel waits.

The bandwidth note set the ceiling by the bytes you move on one GPU. The same ceiling exists one rung up, where the bytes cross NVLink or NVSwitch instead. Keep the arithmetic intensity high. Cross a link only when the math needs data that lives on the other side.

Seven collectives cover everything you will write

Collectives are the standard communication patterns across devices, and every parallel strategy is assembled from them. world_size is the number of devices in the group. rank is the device ID, and it runs from 0 to world_size - 1.

• broadcast: one rank sends a tensor to every rank.
• scatter: one rank sends a different slice to each rank.
• gather: one rank collects the values from every rank.
• reduce: combine the values across ranks with sum, min, or max, and write the result on one rank.
• all_gather: gather, with the full result landing on every rank.
• reduce_scatter: reduce across ranks, then send slice i to rank i.
• all_reduce: reduce, then all_gather, so every rank holds the reduced result.

The names decode themselves. Reduce combines values across ranks and gather collects them onto one rank. Scatter splits the outputs across ranks, and all means every rank ends holding the result.

Four ranks each build t = [0, 1, 2, 3] + rank.

t = [0, 1, 2, 3] + rank

Rank 0: [0, 1, 2, 3]
Rank 1: [1, 2, 3, 4]
Rank 2: [2, 3, 4, 5]
Rank 3: [3, 4, 5, 6]

dist.all_reduce(t, op=SUM)

Every rank: [6, 10, 14, 18]

The call changes the tensor in place, and every rank ends with the same vector.

reduce_scatter takes an input whose first dimension is world_size, so with four ranks the shape is [4, N]. It reduces across ranks, then hands slice i to rank i. Each rank keeps a tensor of shape [N]. Run an all_gather after it and you rebuild what all_reduce produces.

NCCL implements the collectives as ring and tree algorithms over NVLink, NVSwitch, and PCIe. When the program starts, the ranks discover the topology and pick their paths. The collectives then launch CUDA kernels that move data GPU to GPU with little CPU work.

torch.distributed exposes all_reduce, reduce_scatter, all_gather, broadcast, and barrier over two backends. Use nccl for GPU collectives and gloo for CPU collectives. Start one Python process for each rank. Call dist.init_process_group with world_size, rank, backend, and an init method. The collectives then run on ordinary tensors.

Measure the collective the way you measure a kernel

The kernels note put a measurement in front of every optimization. The same discipline applies here, with links in place of SMs, and the steps barely change.

• Run a warm-up pass so the kernels load before the clock starts.
• Synchronize the devices before and after the timed region.
• Use large tensors so that fixed overhead does not own the number.
• Divide the bytes moved by the elapsed time to get effective bandwidth.

Estimate the traffic before you read the clock. During an all_reduce, each rank moves about this much.

The factor of 2 counts two passes. The data leaves once for the reduction, then leaves again to reach every rank.

The benchmark ran four ranks with 100,000,000 float32 values on each, which is 400 MB per rank. Those tensors produced the 277 GB/s. Distance from the H100 NVLink peak moves with tensor size, algorithm, overlap, and topology.

One skipped collective hangs every rank

Every rank must call the same collectives in the same order, with shapes that match. reduce_scatter reads its destinations from the first dimension, so slice i goes to rank i. A shape that differs on one rank breaks the call.

If one rank skips a call, the others wait forever. The job does not crash and it does not print a trace. The ranks that did their part sit inside the collective, and the GPUs go quiet.

all_reduce does two jobs at once. It moves the gradients and it synchronizes the ranks. Skip it on one rank during a training step and every other rank blocks at that line.

Point-to-point carries the same contract. send(tensor, dst_rank) names its destination and recv(tensor, src_rank) names its source. Every send needs a recv that matches it, and sends between one pair of ranks keep their order. A send with no matching recv deadlocks the program. recv writes into a tensor you supply, and both calls block in simple code. isend returns a handle for an asynchronous send.

Two tools make a hang findable. barrier holds every process until all reach the same point. Use it to print in rank order, and sometimes for correctness. The gloo backend runs the same collectives on CPU, where debugging costs less.

Three splits, three costs

Data parallelism splits the batch and is the most common choice. Every rank holds a full copy of the model and its own optimizer. local_batch_size is batch_size divided by world_size, and rank r takes its own slice.

Take a deep MLP whose layers each do a [hidden_dim × hidden_dim] matmul and a nonlinearity. The input batch has shape [batch_size, hidden_dim]. Each rank runs the forward pass on its own slice, so the local loss differs across ranks. The backward pass gives local gradients.

An all_reduce with SUM over each parameter gradient, divided by world_size, makes the gradients identical again. Then every rank takes the same optimizer step. The parameters start equal because every rank uses the same init and the same RNG seed. They stay equal because the gradients synchronize at every step.

The cost is a full copy of the model and its optimizer state on every rank. One all_reduce for every gradient rides on top of that, at every step.

Tensor parallelism splits the model itself. You need it when the model does not fit on one GPU at batch size 1. local_num_dim is hidden_dim divided by world_size, and each rank holds a [hidden_dim × local_num_dim] shard of every layer.

The activations x start identical on every rank with shape [batch_size × hidden_dim]. Each rank computes local_x = x @ local_W, applies the nonlinearity, and holds a [batch_size × local_num_dim] piece. An all_gather concatenates the pieces back into x, and the next layer repeats the pattern. The backward pass runs the same idea in reverse with reduce_scatter and all_reduce.

Each rank stores 1/world_size of the parameters, which is how the width grows past one GPU. The bill is an all_gather inside every layer. The parallelism note kept tensor parallelism inside a node, and this bill is the reason. Only NVLink is fast enough to carry a collective at every layer.

Pipeline parallelism splits the layers by depth. With world_size = 2, a 4-layer MLP puts layers 0 and 1 on rank 0. Layers 2 and 3 go to rank 1. Rank 0 runs its layers and sends the activations forward. Rank 1 receives them, runs its layers, and stores the outputs.

Run that naively and rank 1 waits while rank 0 processes the whole batch. Then rank 0 waits while rank 1 finishes. Those gaps are the bubble.

Microbatches fill the bubble. A batch of 128 becomes 4 microbatches of 32. Rank 0 works on microbatch k+1 while rank 1 works on microbatch k. The last stage computes the loss and sends the gradients back. Each earlier stage backpropagates through its layers and passes them further back.

Other axes exist too, such as the sequence length inside attention. Each of the three splits turns a memory problem into a communication problem.

The Builder Test

Count the bytes one training step sends. In data parallel that is one all_reduce for each parameter gradient. The estimate above gives the bytes for each. Sum them.

Then time the step with the devices synchronized and divide. Compare the effective bandwidth against the link your plan assumed. A rate far under that link points at small messages, a bad algorithm choice, or communication that never overlapped compute.

Idle bubbles, all_reduce time, uneven shards, and small microbatches each erase the gain. Fix the one your measurement names before you add GPUs.

What Carries

More GPUs do not make training faster on their own. A cluster is worth what its schedule keeps in motion, and collectives are what the schedule is made of.

Name the wall before you choose the split. If memory is the wall, shard the state. A matrix too large for one GPU means splitting the operation. A batch that divides cleanly starts with data parallelism.

The cluster runs now. The next question is how large a model and how many tokens to buy with it.

Compute, memory, communication bandwidth, and batch size compete inside one training run. The split you choose is correct only when it answers whichever one runs out first.

One GPU cannot train a frontier language model. The model does not fit, the data does not fit, and the compute takes too long.

Four resources compete, and batch size is the one people forget

Model needs grow faster than GPUs do. A large model strains a single GPU in two places. Billions of weights do not fit in its memory. Long sequences and deep networks build intermediate tensors that do not fit either.

Batch size is the fourth resource, and some splits spend it. You fix that supply when you plan the run, so what one split takes another cannot have.

The unit of compute is now the data center. Near-linear scaling is the goal. Double the GPUs and you want about double the capacity and throughput.

Where a split runs decides what it costs. A cluster has three rungs.

• A node holds several GPUs, often eight, and very fast NVLink or NVSwitch links connect them.
• Across nodes, a fabric such as InfiniBand carries the traffic, slower than NVLink and higher in latency.
• Past a few hundred GPUs, the traffic crosses switch tiers that are slower again.

An algorithm that talks constantly belongs inside a node, where the bandwidth is high. Across nodes, the same algorithm has to justify every byte it sends.

Allreduce equals reducescatter plus allgather

Five collectives show up everywhere in distributed training, and each moves a known amount of traffic.

• Allreduce reduces data across all ranks, for example with a sum, and gives every rank the same result. The bandwidth cost is about twice the data size.
• Broadcast copies the input from one rank to all ranks, at about one times the data size.
• Reduce takes data from many ranks and leaves the reduced result on one.
• Allgather starts with one chunk on each rank and ends with the full concatenation on every rank.
• Reducescatter reduces the chunks across ranks, splits the result, and leaves each rank holding its own shard.

Run a reducescatter and then an allgather and you have done an allreduce. In a bandwidth-limited regime the two paths cost the same. The same bytes move either way, so sharding moves that traffic in two pieces. ZeRO and FSDP stand on that equality.

Data parallelism is simple and pays for it in memory

Naive data parallel SGD copies the model and splits the batch. The global batch is B and the cluster has M GPUs, so each GPU processes B/M examples. Each GPU runs forward and backward, computes its gradients, and allreduces them. Every GPU then holds the full gradient and applies the same update locally.

Compute scales well. Communication for each step is about twice the number of parameters, and a large enough batch hides it behind compute.

Every GPU also stores a full copy of the parameters, the gradients, and the optimizer state. Adam-style training keeps five items for each parameter.

The bf16 parameter and the bf16 gradient take 2 bytes each. The fp32 master weight, the first moment, and the second moment take 4 bytes each. The bill is about sixteen bytes for each parameter.

The accounting note already charged that bill. Sharding starts here because this is where the waste sits. A 7.5B model on 64 accelerators can need about 120 GB of parameter-related memory. In naive data parallel most of that holds copies of the same numbers.

Every GPU computes the same update from the same state. Does every GPU need to hold that state?

ZeRO removes the copies without adding bandwidth

ZeRO means Zero Redundancy Optimizer, and each stage deletes one kind of copy.

Stage 1 shards the optimizer state and leaves the parameters and gradients replicated. Each GPU computes gradients on its mini-batch. A reducescatter sends each summed gradient shard to its owner. The owner updates its own shard, because it holds the optimizer state for that shard. An allgather then rebuilds the full updated parameters on every GPU.

Reducescatter plus allgather costs the same bandwidth as the one allreduce it replaces. Stage 1 adds almost nothing to the wire and divides the optimizer state memory by the number of GPUs.

Stage 2 keeps that and shards the gradients too. Each layer produces its gradients during the backward pass, and a reducescatter sends them to their owners at once. The GPU then frees the gradient buffers it does not own.

After the backward pass each GPU holds one gradient shard and one optimizer-state shard, with the parameters still replicated. The GPUs update their shards, and an allgather restores the full parameters. Traffic stays near twice the parameters for each step, in smaller and more frequent messages. Gradient memory now falls by that same factor.

Stage 3, also called FSDP, shards the parameters as well, and no GPU holds the whole model at one time.

• Forward: allgather the parameter shards for one layer, run that layer on local data, then free the full parameters.
• Backward: allgather the parameters again if needed, compute the gradients, and reducescatter them so each owner keeps its shard.
• Then the owners update their shards and free the full parameters and the temporary gradients.

Communication for each step rises to about three times the parameters, against two for simpler data parallelism. Overlap is what makes that acceptable. While the GPU computes layer k, the network reads the parameters for layer k+1.

Memory for each GPU lands near the minimum. You pay in traffic and you get a model that fits.

Data parallelism runs out when the batch cannot grow

Replicas need batch to feed them. You cannot run more replicas than the global batch size allows. Past the critical batch size, a bigger batch returns smaller and smaller gains.

Sharding does nothing for activation memory at Stages 1 and 2. Stage 3 relieves the parameters and the optimizer state, and activations still dominate at long sequences.

Activation memory climbs through the forward pass and falls through the backward pass. The peak usually sits in the middle of the backward pass.

Two terms stay large in a rough per-layer view. One grows with sequence times batch times hidden size, for the MLP and the pointwise operations. The other grows with sequence squared times batch, for attention.

If the batch cannot grow and the activations are large, data parallelism is finished. The next split must cut the model itself.

Pipeline splits depth and tensor parallel splits width

Pipeline parallel cuts the network by depth. GPU 0 takes the early layers, GPU 1 the middle layers, GPU 2 the later ones. Activations flow forward from stage to stage, and gradients flow back from the last stage to the first.

With one microbatch most GPUs sit idle, and utilization lands near one over the number of stages. Microbatching fills the pipe. GPU 0 sends microbatch 0 forward and starts microbatch 1 at once.

The bubble fraction scales as (S - 1) / M, where S counts stages and M counts microbatches. M must be much larger than S to keep the bubble small.

Pipeline parallel buys memory. Each GPU stores only its own layers, and it ships point-to-point activations that survive slower links. It spends batch size through microbatches, and efficient zero-bubble schedules are hard to build and to maintain.

Tensor parallel cuts by width. Most transformer compute sits in matmuls, so this split shards the large weight matrices. Each GPU computes a partial result, and collectives assemble the correct activations.

That puts a communication point in every layer. The split needs very high bandwidth, so it stays inside a fast node. Eight GPUs on NVLink or NVSwitch is the usual case.

Placement is priced. From 1 to 8 inside a node, tensor parallel costs 10 to 12 percent of throughput. At 16 or 32 across nodes it can cost 40 to 65 percent, because the links are slower.

Tensor parallel does not spend batch size, and that is why it goes first inside the node. It matches matmul-heavy work and it is sensitive to the hardware topology.

Fit the model first, then buy throughput

A plan that does not fit has no throughput to measure. Set tensor parallel to the number of GPUs in one node, for example TP = 8. If the model still does not fit, add FSDP or Stage 3 across nodes. You can add pipeline parallel as well, or both.

Once the memory fits, add data-parallel replicas to raise the total FLOPs. Data parallelism tolerates slower links, and it is the easiest split to reason about.

Batch size is a hard budget, and both pipeline parallel and data parallelism draw on it. If the batch is small and you want fewer synchronizations, use gradient accumulation.

Activations are the last term to fall. Naive tensor parallel leaves pointwise operations such as LayerNorm and Dropout unsharded. Sequence parallelism shards that work along the sequence dimension, so each GPU handles about sequence over TP tokens. When you need a full-sequence view, an allgather or a reducescatter provides it.

Recomputation is the trade the GPU note already priced, applied here to activations. Flash-style attention never stores the full attention matrices. Tensor parallel, sequence parallelism, and recomputation together drop the activation memory on each device.

Production runs mix all three splits. Megatron-style training spans about 1B to 1T parameters. Tensor parallel stops near 8, pipeline parallel grows with depth, and data parallelism balances batch against compute. That mix reaches about 40 to 50 percent of theoretical GPU FLOPs.

The Builder Test

Name the wall in one sentence before you choose a split. Then say which resource the split spends to buy relief.

• Memory is the wall: shard the optimizer state first, and the traffic bill does not change.
• One matrix is too large: split the operation inside the node, and pay in intra-node bandwidth.
• The batch divides cleanly: add data-parallel replicas, and pay out of the batch budget.

Write that sentence down before you write the config.

What Carries

Parallelism is a response to the thing that does not fit. A split that answers no wall still costs bandwidth.

The plan so far lives on paper. The next step is to measure what the collectives cost on your links.

A fused GeLU runs in 1.1 ms. The same formula written as raw PyTorch math takes 8.1 ms. The hand-written CUDA kernel lands at about 1.8 ms, behind the library it set out to beat. Measure first, then fuse the memory-bound chains, and write a kernel only when the profiler names one.

Measure first or you are guessing

Three traps make a GPU benchmark lie, and each one has a fix that costs nothing.

• The first CUDA call compiles or loads code, so trial one reports startup instead of steady state.
• Python returns as soon as it queues the work, so an unsynchronized timer measures the queue.
• Thermals, operating system activity, and other processes move the number from run to run.

A benchmark that reports the queue makes a slow kernel look free. You then optimize the operation next to it.

One helper answers all three in five steps. Run warmup iterations, synchronize, time several trials, synchronize after each trial, and report the mean. A timing taken any other way is a rumor about the kernel.

The profiler tells you which operation owns the time

The PyTorch profiler splits a workload into Python calls, aten operations, CUDA kernels, and launch overhead. It tells you whether the GPU or the dispatch layer owns the time.

Profile A + B and the shape is clear. The CUDA elementwise kernel finishes in microseconds while aten::add and cudaLaunchKernel own most of the CPU time. Nothing about that operation is worth a rewrite.

Profile torch.cdist and the answer moves. The call breaks into matmuls, elementwise work such as pow and subtract, plus reductions and a square root. The matmuls take more than 70 percent of the time, so the gain lives there and nowhere else.

Nsight Systems puts the CPU threads and the GPU timeline on one screen. NVTX ranges such as forward and backward mark the phases. Two behaviors show up there that the operator table hides.

The CPU runs far ahead of the GPU and queues kernels the GPU executes later. A print of a GPU scalar forces a synchronize, and the CPU then waits for the GPU to drain. Frequent prints and frequent .item() calls in a training loop pay that stall on every iteration.

Python slowness usually does not matter when the GPU owns the time. The exceptions are large pure-Python loops and frequent syncs.

The operator table also shows which of your calls are already fused. A fused built-in appears as one kernel and a naive chain appears as many.

Fusion is where the free speed lives

Fusion combines several operations into one kernel. The intermediate values stay in registers instead of going to global memory. That cuts the traffic and the launch overhead together.

A fast kernel obeys one pattern. Load from DRAM once, reuse the values in registers or shared memory, and write to DRAM once. Every extra trip to global memory is time the arithmetic never sees.

The naive GeLU launches a separate kernel for each multiply, add, tanh, and cube on the tensor. Every kernel reads the tensor from global memory and writes it back. The built-in F.gelu does all of that math inside one kernel with the intermediates held in registers.

The GPU note gave the rule that explains the gap. Arithmetic intensity is FLOPs for each byte moved, and compute grows faster than bandwidth in every hardware generation. An elementwise chain carries almost no arithmetic for each byte, so it waits on memory the whole way.

Fusion buys its speed by cutting trips, and the arithmetic stays exactly the same. An operation whose arithmetic intensity you cannot state is an operation you cannot correct.

The same rule sorts the rest of your model. Large, well-tiled matmuls run compute-bound. Elementwise operations, small matmuls, and work with a poor access pattern run memory-bound. Fusion pays on all three.

A custom kernel in CUDA C++ or Triton runs the same GeLU at about 1.8 ms. torch.compile on the naive code runs it at about 1.47 ms and generates the fused Triton kernel itself. Someone tuned the library kernel for these shapes on this hardware. A first hand-written version starts with none of that tuning.

Triton buys most of CUDA's speed in Python

The CUDA version makes you write both halves by hand. Mark the kernel __global__, compute the global index from the block and thread ids, then bounds-check and store. Set CUDA_LAUNCH_BLOCKING=1 while you debug so the error surfaces at the line that caused it.

Triton moves that work into Python at the block level. You write one program instance for each block, and Triton handles coalescing and the low-level details.

pid = tl.program_id(0)
offsets = pid * BLOCK_SIZE + tl.arange(0, BLOCK_SIZE)
mask = offsets < num_elements
x = tl.load(x_ptr + offsets, mask=mask, other=0)
# Compute GeLU on x as vector math
tl.store(y_ptr + offsets, y, mask=mask)

The Triton GeLU runs at about 1.84 ms, which sits next to the CUDA C++ kernel. The code reads like vectorized Python math, which makes it cheaper to change.

Softmax is harder than GeLU because it needs a row-wise reduction. Subtract the maximum, exponentiate, sum, and divide. The simple design gives one row to one block, with as many blocks as rows. BLOCK_SIZE is a power of two, no smaller than the number of columns.

On a large matrix that design loses. The naive PyTorch softmax takes about 3.7 s and the built-in takes about 1.5 s. torch.compile takes about 1.3 s and the simple Triton kernel about 1.9 s. The compiler searches tilings and fusion choices that a first draft in Triton does not.

torch.compile traces the code into a graph, fuses the elementwise chains, and picks kernels for your shapes and hardware. Most of the fusion gain arrives without a kernel written by you.

Write a custom kernel only when the profiler names it

The accounting note ran one triage on a budget, and the same triage runs on a profile. Spend the effort where the share of the time is large enough to repay it.

A hand-written kernel earns its place when the libraries cover nothing like the pattern. The other case is a tight inner loop the profiler flags, where the compiler misses hardware behavior that matters.

The numerical check is the part people drop. A kernel that is faster and wrong costs more than the operation it replaced. Run the simple baseline next to the kernel on the same input, and compare time and output.

The Builder Test

Profile one training step before you change a line. Write down the top three operations and the share of the time each one owns. The operator breakdown answers which operation, and the timeline answers whether the GPU waited.

If that list surprises you, the optimization you had planned was a guess. Start at the top of the list instead. Re-profile after the change, so the next decision has a number under it. I rank operations by their share of the time before I open an editor.

What Carries

A guess about the hotspot costs more than the measurement that names it. Memory, compute, communication, and data each call for a different tool.

Name the constraint first, then pick the tool that attacks it. One GPU is busy now, so the next constraint appears when the model needs many of them.

Compute grew faster than memory bandwidth, so the bytes you move set the ceiling on every kernel. A modern GPU spends most of its time moving data to and from global memory. Count the bytes a kernel moves before you count its FLOPs.

FlashAttention is the proof. It computes exact attention and never writes the n by n score matrix to global memory. The math stays the same. The traffic drops.

Bytes moved set the ceiling

Progress in language models comes as much from GPU hardware and parallelism as from new model ideas. Language models obey scaling laws, so more compute and more data usually improve results.

For decades that compute arrived as faster single cores. Dennard scaling and Moore's law delivered more transistors, smaller and faster and lower in power.

Then single-thread gains flattened. Transistor counts kept climbing while clock speeds and single-thread performance stopped climbing. Deep learning took its gain from parallelism instead, running more operations at the same time.

The parts of the machine did not scale together.

• Host-device links, PCIe and NVLink, improved slowly.
• Global memory bandwidth improved from GDDR to HBM2E, and the improvement stayed modest.
• Compute, and matmul FLOPs in particular, grew by orders of magnitude.

Older GPUs can be FLOP-bound. Modern GPUs are often memory-bound, and the memory hierarchy is where that shows up.

The hierarchy is strict. Memory near the core is fast and small. Memory far from the core is slow and large.

• Registers are per-thread storage and the fastest memory on the chip.
• Shared memory and L1 sit on the SM, and every thread in a block reads them. They are the right home for tiles and reused data.
• L2 is shared by all SMs, slower than shared memory and faster than DRAM.
• Global memory is off-chip HBM. It is large and much slower than anything on-chip.

If a kernel reads and writes global memory without pause, the SMs wait. Bandwidth decides the result and the arithmetic units sit idle.

Arithmetic intensity is the FLOPs a kernel does for each byte it moves. The roofline model reads it off two axes. The horizontal axis is FLOPs per byte moved. The vertical axis is the achieved FLOPs per second.

Low intensity puts you on the slope, where throughput rises as intensity rises and bandwidth owns the result. High intensity puts you on the flat roof, where peak compute owns it.

The accounting note priced a run before the launch. The roofline asks the same question of a single kernel.

A kernel falls below the roof for four reasons.

• It makes too many reads and writes against global memory.
• Its threads scatter across the address space.
• Its tiling is weak, so reuse stays in global memory.
• Its sizes do not divide cleanly into tiles.

A warp that splits pays for both paths

Manufacturers build a GPU for throughput. It carries many simple compute units with little control logic for each.

An SM is the unit of scheduling and control. It owns registers and shared memory, and it runs many warps at once to hide stalls. When one warp waits on a load, the SM runs another.

An SM holds streaming processors, which are simple arithmetic units. It also holds tensor cores, which are built for matrix multiplication. Tensor core throughput on matmul sits far above the general-purpose FLOP rate.

A model must be matmul-heavy to use the full capability of the chip. An A100 carries more than 100 SMs, each with many streaming processors and tensor cores.

The GPU arranges work into three units.

• A thread is the smallest unit of work, with its own registers and local state.
• A warp is 32 threads running the same instruction on different data at the same time.
• A block is a group of threads scheduled onto one SM, able to share memory and synchronize.

Blocks map to SMs, and warps run inside blocks under SIMT. A warp runs one instruction at a time, so threads that take different branches cannot run at once.

Put a branch inside a warp: if thread_id < 4 do A, else do B.

• Threads 0 to 3 run A while the others stay idle.
• Then threads 4 to 7 run B while the first group stays idle.

The warp runs both paths in sequence and pays the time of both. Every conditional that splits a warp is expensive for that reason. The cost never shows up in a FLOP count.

Four habits recover most of the lost speed

Every habit here does one of two things. It raises arithmetic intensity, or it cuts the bytes moved. Check any new habit against that pair before you spend a week on it.

Lower precision

Fewer bits for each number means fewer bytes moved. Lower precision also often buys more math operations for each cycle.

Take ReLU, x = max(0, x). In float32 the GPU reads 4 bytes and writes 4 bytes, which is 8 bytes for each element. In float16 it reads 2 and writes 2, which is 4 bytes for each element.

The operation count does not change and the traffic halves. Arithmetic intensity doubles, and the tensor cores come into range.

Operator fusion

Naive GPU code launches one kernel for each small operation. Each kernel makes its own round trip to global memory.

Write y = sin(x)² + cos(x)² the naive way.

sin → write s
cos → write c
square s → write s2
square c → write c2
add → write y

Every step in that chain reads global memory and writes it back. The fused kernel loads x once, computes in registers and shared memory, and writes y once.

Global traffic drops and speed rises. torch.compile fuses many chains like this one without help from you.

Recomputation

Backprop stores intermediate activations so the backward pass can reuse them. Those activations live in global memory, and reading them back costs more than making them again.

Three stacked sigmoids show the cost. The naive forward computes s1, s2, and s3, then stores all three and the output. The naive backward reads all three again from global memory.

Recomputation drops the internal sigmoids in the forward pass. The backward pass rebuilds them from x inside the kernel, then computes the gradients.

You spend compute to buy traffic. Compute is cheap on a modern GPU and bandwidth is scarce, so the trade usually pays.

This is checkpointing under another name, and speed is the reason to reach for it here. The same trade returns in the parallelism note, where activation memory is the last bottleneck standing.

Coalesced access

Global memory moves data in bursts. Fetching element 0 pulls a full aligned chunk, and the start of that transfer is the expensive part. Bytes next to it are cheap once the burst runs.

Threads in a warp that read nearby addresses let the hardware fold those reads into a few bursts. Threads that read scattered addresses start many bursts and waste the bandwidth they paid for.

Coalesced access holds the effective bandwidth high. Non-coalesced access collapses it.

Check the layout before you check the loop. A thread that walks across a row can still be strided in memory, and the warp loses its bursts.

Tiling moves the reuse onto the chip

Matmul is the workload the GPU is built for. A naive kernel reads the same values of A and B from global memory many times over.

Tiling moves that reuse onto the chip.

• Split A and B into tiles.
• Load one A tile and one B tile into shared memory.
• Accumulate partial results for a tile of C.
• Repeat for every tile pair that feeds that C tile.

Each global value now loads far fewer times. Inside a tile the reuse happens in shared memory, and global reads drop by about the tile size factor. The work moves onto fast on-chip memory.

Four things limit the tile you can pick.

• Shared memory size caps how much of A and B a block holds.
• The warp structure fixes how threads cover the tile.
• Coalescing decides whether the tile loads ride full bursts.
• Divisibility decides whether the last tile is full or ragged.

Miss any one of the four and the tile stops paying.

Matmul speed looks wavy because sizes must divide cleanly

Throughput generally rises as the matrix grows. The curve still carries dips and waves as it climbs.

Sizes that are multiples of the tile, warp, and burst sizes keep the hardware busy. Sizes that are not give you partial tiles, idle threads, and extra memory transactions. An awkward size runs much slower than a clean size beside it.

Tile count against SM count adds its own dip. Each tile maps to a block and then to an SM. A matrix that needs 98 tiles on a GPU with 108 SMs runs in one wave.

A matrix that needs 120 tiles runs 108 tiles first, then 12 more in a second wave at low occupancy. Throughput drops at exactly those sizes.

Burst boundaries add another dip. Tile widths that line up with DRAM bursts let a row fit into a few of them. Add one column, the rows cross a boundary, and the number of bursts doubles.

These effects combine, and the measured curve waves. Round your dimensions to the hardware before you blame the kernel.

FlashAttention keeps the math exact and refuses to write the n by n matrix

Standard attention runs three steps.

For sequence length n, S is n by n. Storing S or the softmax weights costs O(n²) memory. Moving that matrix in and out of global memory is the real bill.

The math stays O(n²) for general attention. The traffic does not have to. FlashAttention keeps HBM access far below the naive version and returns the same numbers, so nothing downstream changes.

Softmax is row-global, which looks like a barrier to tiling. Online softmax removes the barrier.

Stable softmax takes four steps.

• Take the max of the row.
• Compute exp of x minus that max.
• Sum the exps.
• Divide by the sum.

Online softmax walks the row in chunks instead. It carries a running max m and a running sum d over the prefix it has seen. Each new chunk updates both, and a larger max rescales d.

The kernel can now stream over the score tiles. The normalization updates while the stream runs, so the full row never lands in global memory.

The forward pass combines tiled matmuls, online softmax, and on-chip storage.

• Partition Q, K, and V along the sequence dimension.
• For each query tile, load the Q tile and one K tile into shared memory.
• Compute the score tile, then update the running max and running sum for each row.
• Move to the next K tile.

After the last K tile the normalization is known. The kernel forms the output with the intermediates still on chip.

The backward pass is harder, because naive gradients need the softmax outputs, and those are n by n. FlashAttention recomputes them instead.

• Loop over the tiles again.
• Rebuild the local scores and softmax values from Q, K, and V with the forward formulas.
• Compute the gradients for that tile.
• Discard the intermediates.

The pass spends more compute and cuts both traffic and storage.

Tiling, online softmax, recomputation, and matmul hardware turn attention from a memory problem into a workload that fits the chip.

The Builder Test

Pick the operation you want faster. Divide its FLOPs by the bytes it moves.

Name the habit that moves that number: precision, fusion, recomputation, coalescing, or a tile that fits the hardware. Write the name down before you write any code.

If no habit raises the intensity, the kernel already sits at the roof. Look for the time somewhere else in the step.

What Carries

A hardware-efficient algorithm is a memory-efficient algorithm. The roofline tells you which half of the machine you are fighting.

A GPU runs many threads at once and stays fragile in two places. A split warp pays for both paths, and a scattered access pattern wastes the bursts it starts.

You know the habits now. The next question is which of them is worth a kernel you write yourself.

Mixture of experts buys more parameters at the same FLOPs for each token. It charges the gain back in routing discipline, load balance, and all-to-all traffic. For a fixed FLOP budget an MoE design is usually the correct choice. The forward pass is the easy part of it.

Replace each dense MLP with many smaller MLPs called experts. Add a router that picks a few experts for each token. The FLOPs for each step hold and the parameter count climbs.

More parameters, the same FLOPs for each token

The name misleads. Nobody hand-assigns a coding expert or an English expert. The experts sit inside the MLP block, and the router learns from data which few of them a token wakes. Attention stays dense.

Give each expert the size of the dense MLP and route each token to one expert. The FLOPs for each forward pass match the dense model. The parameter count rises with the number of copies. The gain lands only if those parameters help the model store more facts and patterns.

Routing must stay sparse to keep that arithmetic. Waking every expert for every token raises the FLOPs too far to pay.

Fedus and coauthors reported in 2022 that more experts give lower loss at fixed FLOPs. Downstream scores moved with it. OLMo-MoE from AI2 reports the same pattern in recent models. By 2025 the advantage over dense models at fixed training FLOPs is clear.

The tokenizer note set the frame: efficiency decides how much scale you can afford. MoE is that lever one level up, and it buys more parameters for each FLOP you already pay.

Upcycling is the cheap way in. Start with a trained dense model, copy each MLP into several experts, and add a router. Perturb the copies if you want, then keep training as an MoE. If training stays stable, you keep what the dense model learned for a small amount of extra training. MiniCPM and Qwen report strong results from this path.

The gains are largest where multi-node training and complex parallelism are already necessary. Below that size the extra complexity costs more than it returns. If you can carry the complexity, MoE is cost-effective.

The hard part is shipping every token to its expert and back

MoE looks simple on slides and messy in production. Top-k selection is discrete. Gradient descent prefers smooth choices, and you cannot backprop through a choice. Naive training goes unstable and leaves many experts unused.

The systems half is heavier. MoE adds expert parallelism on top of whatever split the cluster already runs. The cluster shards the experts across devices, so each device holds one or a few. After routing, every token travels to the devices that hold its chosen experts. The experts run, the outputs come back, and the layer combines them. That exchange is all-to-all, and it happens at every MoE layer of every step.

On a multi-node cluster the exchange can cost a few milliseconds for each layer. It is the main trade against the compute gains and the memory gains. An MoE layer spends fewer FLOPs for each token and more bytes on the interconnect. The ceiling follows the bytes.

Modern kernels and libraries fuse many small expert matmuls into larger sparse operations. Small experts still run efficiently that way. The repair works on the compute side. The traffic stays.

Token-choice top-k routing won because the router must stay cheap

Each token arrives at the layer as a hidden state x, the residual input to the MLP. The router holds one vector e_i for each expert, kept separate from the expert weights.

• Score. Each expert gets a score equal to the dot product of x and its vector e_i.
• Normalize. A softmax over the experts turns those scores into s_i.
• Select. Take the top k by s_i. The gates g_i equal s_i for the chosen experts and zero for the rest.
• Compute. Run the chosen MLPs on x and weight each output by its gate. Add the products and send the sum into the residual stream.

That is the entire router: a linear layer, a softmax, and a top-k. Router compute that grows eats the savings the layer exists to create. The learning signal is weak and indirect, so more router machinery does not pay. A small router is also easier to stabilize and debug.

k is a hyperparameter with a price. At k equal to one you wake a single expert. At k equal to two you buy redundancy and exploration, and the MLP compute roughly doubles.

Earlier work argued for two so the router can compare experts instead of committing early. A larger k costs more FLOPs and more communication. Fine-grained experts shrink each expert, so a larger k stays affordable.

Five other routing designs lost, each for a named reason.

• Expert choice. Each expert picks its top-k tokens. That balances the load and swaps the question to what fits this expert.
• Global assignment. A matching problem balances the load exactly. Solving it costs too much once the model is large enough to need it.
• Hash routing. Sending tokens by hash ignores the semantics and still beats dense models. Even a crude partition across many MLPs helps.
• RL routing. Researchers tried it early because routing is discrete. It costs more than the simple recipe and does no better, so they stopped.
• Stochastic routing. Noise on the router logits pushes tokens toward other experts. It helped less than the balancing heuristics, and most systems dropped it.

Most strong open MoE models route this way, including OLMo-MoE, DeepSeek, Qwen, Mixtral, and the Google MoE variants. The recipe under all of them is the same: token-choice top-k routing plus explicit balancing.

Without balancing, training collapses onto a few experts

Without a balancing term, most tokens route to a few experts. Those experts get good at every task and the rest never learn. OLMo-MoE ran the ablation. Two experts take half the tokens and the others stay unused. Validation loss rises, and you paid for parameters that do no work.

The goal is to spread the traffic so every expert learns and no device turns into a bottleneck. The Switch-style auxiliary loss builds two vectors over the experts. f(i) is the fraction of tokens that reach expert i after top-k. p(i) is the fraction of router probability mass on expert i before top-k.

The loss is the dot product of f and p across the experts. An expert that already takes many tokens gets its router probability pushed down. Variants apply the loss for each expert and for each device.

DeepSeek v3 balances without an explicit auxiliary loss. It keeps a bias b_i for each expert and counts the tokens each expert took after every batch. An underused expert gets a higher b_i and an overused one gets a lower b_i.

Routing adds b_i to the score before the softmax or sigmoid. It stays out of the final gate weights. Tokens move toward the underused experts and the weights the layer applies do not change. A sequence-wise balancing loss sits on top. It corrects imbalance inside one sequence at inference rather than only across batches.

The router softmax goes numerically unstable, so systems compute it in float32. A z-loss on the router logits penalizes the log-softmax normalizer. Small logits mean fewer loss spikes.

The extra parameters overfit small fine-tuning sets. DeepSeek answers with very large supervised fine-tuning data, so the MoE choice reaches back into what you collect.

Fine-grained experts buy more than shared experts

The naive build copies the whole MLP into every expert, and the parameter count climbs fast. Fine-grained experts split the MLP expansion dimension instead. That dimension is often four times the hidden size. Cutting it into narrow pieces gives you many more experts while the compute for each token stays low.

Shared experts run for every token and hold the patterns that are useful everywhere. That is also their price. Their compute lands on the bill for every token. A fine-grained expert costs you only when the router picks it.

DeepSeek made both moves popular. Ablations in DeepSeek and OLMo-MoE show that more fine-grained experts improve the loss and the benchmarks. Shared experts help in some settings and are not necessary in all of them.

Early Google systems ran 8 to 16 experts in each layer with 1 or 2 active. GShard, Switch, and ST-MoE all sat in that band. DeepSeek and the labs that followed run dozens of fine-grained experts. Most of them keep at least one shared expert and several active for each token.

Spend the fine-grained split first. Add a shared expert when a measurement asks for one.

Token dropping makes inference non-deterministic at temperature zero

Experts and devices carry a capacity limit for each batch. When an expert receives more tokens than it can hold, some systems drop the overflow. Those tokens skip that expert, and the model does not use it for them.

The drops depend on what else is in the batch. Another request landing in the same batch changes which tokens the system drops. The same prompt at temperature zero then returns a different answer. A test you ran alone does not predict what a user gets inside a full batch.

I do not promise a reproducible answer from a system that drops tokens. Reproducibility turns into a property of how you batch. Measure the drop rate on a production-shaped batch before you make that promise.

The DeepSeek line changed routing and balancing while the layer types held

DeepSeek MoE v1 held about 16B parameters with about 2.8B active. Each layer carried 64 fine-grained experts and 2 shared experts, with several active for each token. Routing was token-choice top-k, with the softmax before the top-k and a weighted sum of expert outputs. Balancing was an auxiliary loss for each expert and for each device.

v2 reached about 236B total and about 21B active on the same MoE core. It added device top-m routing. The router picks a small set of devices for each token, then picks experts inside them. That cuts the traffic. Communication balancing losses came with it.

v3 reached about 671B total and about 37B active. Routing normalizes the gates so the expert outputs sum to one. It uses sigmoid gating in one part and keeps device top-m. Balancing moved to the online bias for each expert plus the sequence-wise loss.

From about 16B to about 671B, the changes sit in routing, balancing, and systems tuning. No new layer type appears. Device top-m is the change that attacks the traffic directly, and it stayed from v2 onward. Read the two parameter counts together, because the total sets the memory and the active count sets the FLOPs.

MLA answers the KV cache with a smaller cache

The architecture note named the KV cache as the reason attention turns memory-bound during decode. MLA is the answer DeepSeek v3 gives to that bill. It cuts KV cache memory and the FLOPs do not rise.

Standard attention caches the full keys and the full values for each token. MLA compresses the hidden state h_t into a smaller latent c_t.

The layer caches c_t alone. When it needs keys and values, it up-projects them.

Written that way, MLA adds matmuls. Merge the up-projection into the projections next to it and the count stays about the same. RoPE makes the merge harder, because it rotates Q and K between the projections. The MLA design keeps RoPE clean by rotating only the parts it leaves uncompressed.

Multi-token prediction adds signal without a new layer

Standard training predicts the next token. Multi-token prediction uses the hidden state at each position to predict further ahead. An extra head or a small layer does the work. DeepSeek v3 adds one head for one step ahead, which gives each sequence more learning signal and can improve efficiency.

The Builder Test

Read four numbers on the same window: expert load, routing by domain, quality by domain, and communication overhead. Routing by domain is the share of each domain that lands on each expert. One of them alone hides the failure. Expert load can look flat while one domain rides two experts. Average quality can look fine while that same domain gets worse.

Then change one routing knob and read the four again. If the loss improves while the drop rate rises, the win came with a cost the loss curve cannot show. When communication overhead climbs and quality holds, you bought the same model for more money.

What Carries

Conditional capacity is real. You pay for it at the router. Instrument the router first, because it decides which parameters wake and how many bytes cross the interconnect.

Routing turns compute into traffic, so the next question is what the hardware charges to move bytes.

Show me the next open model and I will guess its architecture before I read the paper. Decoder-only, pre-norm, RMSNorm, SwiGLU, RoPE, no bias terms, and a width-to-depth ratio from a narrow band. The guess lands, because those choices train stably and serve cheaply.

You can guess a new open model's architecture and be right

Many labs trained many models. The designs that come back run after run are the ones that survived measurement. Copying them is the cheapest thing a builder does, and knowing why they won is the part that pays.

Two bills drive the convergence. A design that spikes during a long run costs you the run. A design that drags a large KV cache costs you every request you serve.

Four defaults moved away from the original transformer. Post-norm became pre-norm, and additive position embeddings became RoPE. ReLU inside the MLP became SwiGLU, and most linear layers lost their bias terms.

The rest did not move. Token and position information enters a stack of blocks, each carrying attention, an MLP, residual connections, and normalization. A serial block runs attention and then the MLP, each part with its own pre-norm and its own residual add. A final softmax over the vocabulary gives next-token probabilities.

The hyperparameters cluster the same way. Families keep d_model equal to n_heads times d_head, which keeps the attention sizes predictable. Too many heads leave d_head small, and a small d_head limits expressiveness. The pretraining loss follows the total parameter count more than the depth, so the shape sits in a broad band. At fixed compute, more depth helps some downstream behavior.

Vocabulary size shifted between generations. English-focused models used 30k to 50k tokens, and newer multilingual and production models use 100k to 250k. The larger vocabulary shortens sequences across scripts. That is the tokenizer note's compression ratio, arriving here as an architecture number.

Pre-norm keeps the residual path close to identity

The original transformer ran a sub-block, added the residual, then normalized. Pre-norm reverses that order. It normalizes the input to the sub-block, then adds the sub-block output to the residual stream.

The order decides what the residual path carries. Under pre-norm it stays near identity, and gradients flow more cleanly through deep stacks. Post-norm still trains, and it needs more careful warmup and more careful tuning to avoid loss spikes. Check the warmup first when a deep stack spikes early.

Some models add a normalization after the sub-block and leave the residual stream itself un-normalized. That double-norm pattern is a stability tool in the largest runs.

RMSNorm drops the mean and the bias and loses nothing

LayerNorm normalizes by the mean and the variance, and it carries both a scale term and a bias term. RMSNorm normalizes by the root mean square and keeps a learned scale only. Dropping the mean subtraction and the bias costs no quality. RMSNorm is cheaper and faster, which is why it dominates recent large models.

The same reasoning strips bias terms out of most linear layers. The parameter count falls, the kernels get simpler, and stability improves in many large runs. Bias terms add a small amount of expressive power, and that gain does not pay for its cost. Those edits are cheap, and together with pre-norm they buy stability and runtime.

Gated MLPs won on repeated measurement

Early transformers used ReLU, and GPT-style models popularized GeLU. Across many training runs, GLU variants reached lower loss and better downstream results at similar parameter counts. SwiGLU is the strongest of them, and gated MLPs are the default now.

The gated MLP splits the input path in two.

• Compute a main projection and a gate projection.
• Apply a nonlinearity such as GeLU or Swish to the main path.
• Multiply the main path and the gate path elementwise.
• Project the result back to the model dimension.

GeGLU and SwiGLU are the common variants, and SwiGLU uses Swish for the gate. The gate works as a learned filter over the hidden dimensions, deciding which channels survive the multiply.

The extra projection costs parameters, so the hidden width comes down to pay for it. A non-gated MLP uses d_ff = 4 × d_model. A GLU MLP uses d_ff near (8/3) × d_model, about 2.66 times the width. Three projections at 2.66 times the width cost what two projections at 4 times the width cost. The parameter count holds, so the comparison between the two MLP types stays honest.

RoPE puts relative position inside the dot product

Position methods varied, and they include sinusoidal absolute embeddings, learned absolute embeddings, relative-bias methods, and ALiBi. Those methods add a position vector at the bottom of the network. RoPE became the default for most dense models, and it never touches the bottom of the network.

RoPE leaves that formula alone and turns its inputs. It pairs up dimensions of the query and key vectors and rotates each pair by a position-dependent angle. The frequency differs for each pair. The dot product between a rotated query and a rotated key then carries the relative position between them. That placement makes RoPE compatible with context extension methods.

Stability tricks exist because large runs break without them

Gradient spikes and numerical problems around softmax end large training runs. The softmax exponentiates its inputs, so one extreme logit is enough to overflow. Two softmax sites carry most of that risk: the final vocabulary softmax and the attention softmax inside each block.

z-loss covers the output side. Add a small penalty on (log Z)², where Z is the softmax normalizer. The penalty discourages extreme logits and keeps the softmax in a safer numeric range.

QK norm covers the attention side. Normalize the queries and the keys before the dot product, and the attention logit scale stays under control. The overflow risk falls, and a larger learning rate sometimes becomes usable. Direct soft caps on the attention logits exist, and their results are not consistent.

One knob from the regularization era survives, and its job changed. The overfitting intuition does not transfer to pretraining. The data is very large and the model sees each example about one time. Dropout left most large runs.

Weight decay stayed because it helps optimization. It interacts with the learning rate schedule and improves the final training loss. The gain arrives late, after the learning rate decays.

Inference pressure explains MQA, GQA, and hybrid attention

Training is compute-heavy and runs in parallel across the whole sequence. Decoding produces one token at a time, so memory traffic dominates attention. The KV cache stores the past keys and values, so the model does not compute them again at every step.

The cache trades compute for memory traffic. Every step reads the whole cache back, and the cache grows with the context. Decoding becomes more memory-bound the longer the context runs.

MQA shares one set of keys and values across all query heads, which shrinks the cache by a large factor. GQA groups the heads so several query heads share one K/V set. GQA holds quality in most cases and still cuts the memory, which puts it between MQA and full multi-head attention.

Full attention is quadratic in the sequence length, so a very long context needs structure. Most layers now run local sliding-window attention with RoPE. A few layers run global full attention, and those global layers often drop RoPE.

Local layers handle nearby structure cheaply. Global layers mix information across the whole sequence, and only a few layers pay the quadratic cost. Removing the positions in those global layers can improve extrapolation.

Each choice here answers the serving bill, and the serving note is where that bill gets priced.

The Builder Test

Predict what a knob changes before you turn it. Depth, width, context length, and learning rate each need a stated reason and a stated expected effect. Write the prediction down before the run and compare it to what the loss did.

Change one knob at a time. A better loss then has a named cause, and a broken run has one thing to undo.

What Carries

A transformer block edits a stream of token vectors. Attention moves information between positions, and the MLP transforms each position. Normalization and the residual path keep the edits stackable. Every swapped default protects that stacking, during the run or during serving.

The model is the block plus the training recipe plus the data plus the compute budget. Architecture becomes behavior only through training.

The block is settled, so the next lever on capacity is how many parameters run for each token.

A training step costs about six times the token count times the parameter count. AdamW costs about sixteen bytes for each parameter. Those two numbers price a run before you launch it.

The tokenizer note settled what a string costs in tokens, which makes it the first lever on this bill. The rest is arithmetic you can do on paper. I do it before I ask anyone for a cluster.

One step costs about six times tokens times parameters

Matmul dominates the cost of a training step. Count the matmuls and you have the whole estimate.

A B × D matrix times a D × K matrix produces B × K output elements. Each element does D multiplies and D adds. The total comes to about 2 × B × D × K.

The units matter when you read a spec sheet. One FLOP is a single floating point add or multiply. FLOPs is a total count of them. FLOP/s is a rate.

Take a two-layer model. X is B × D, W1 is D × D, and W2 is D × K. Forward computes H1 = X × W1, then H2 = H1 × W2. That first matmul costs 2 × B × D², and the second costs 2 × B × D × K. Together they come to about 2 × B × (D² + D × K).

D² + D × K is the parameter count of that model. B is the number of data points. So the forward pass costs about two times data points times parameters.

The backward pass runs two matmuls for every one the forward pass ran. One produces the gradient for the weights and one passes the gradient down to the input. For W2 that is H1 transposed times grad_H2, at about 2 × B × D × K. Then grad_H2 times W2 transposed, at the same cost. W1 needs the same pair over its D × D shape.

Backward totals about 4 × B × (D² + D × K), twice the forward pass. Add the two and one step costs about six times tokens times parameters.

Language models feed batch, sequence, and hidden shapes into batched matmuls. The same weights apply to every token in every sequence. Data points are tokens, which is why the formula reads in tokens and parameters.

AdamW costs about sixteen bytes per parameter

AdamW in mixed precision costs about sixteen bytes for each parameter. Two bytes hold the bf16 weights and two hold the gradients. Four hold the fp32 master copy. Four more hold the first moment, and four hold the second.

A 7B model is roughly 112 GB of state before a single activation. An H100 has 80 GB. One card cannot train that model without sharding or gradient accumulation.

That sixteen-byte bill is the reason sharding exists, and the parallelism notes spend themselves on paying it down.

The optimizer decides eight of the sixteen, four bytes for each moment. SGD keeps no extra state. Momentum, AdaGrad, and RMSProp each keep one extra tensor for each parameter. Adam keeps two moving averages, the first moment and the second.

Answer two questions before you launch

Two sketches decide whether a run is worth asking for.

How long does a 70B transformer take on 15T tokens with 1,024 H100s? Total operations come to 6 × parameters × tokens. Pick the peak FLOP/s for the dtype you run, then apply an MFU guess. 50 percent is reasonable for a well-tuned configuration.

Multiply peak by MFU, then by the GPU count, then by the seconds in a day. That gives operations for each day. Training days are the total divided by that. The MFU guess is the soft part of the sketch. Carry it as a range and check it on the first day of the run.

If the day count comes back too high, the formula names the moves. Cut tokens, cut parameters, add GPUs, or raise MFU. Two of those change the model you get, and two change only the bill.

What is the largest dense model that fits on 8 H100s under AdamW with no memory optimizations? Each card carries 80 GB of HBM. Divide the total by sixteen bytes for each parameter. The answer lands near 40B parameters. Activations, sequence length, and other overhead pull it down from there.

Run the memory sketch first. A run that does not fit never gets to be slow. Leave headroom for activations, because the sixteen bytes cover none of them. If the model overruns the cards, shard it, accumulate gradients, or make it smaller. Each of those three changes the time sketch too.

MFU tells you whether the GPU is working or waiting

MFU is the operations your model needs divided by the operations the card can do in that time.

• Count the model FLOPs by adding the matmuls at two times the product of their dimensions.
• Measure step time.
• Divide operations by time for the measured FLOP/s.
• Divide that by the peak FLOP/s of the dtype you run.

Then read the number. About 0.5 or more is strong for a real system. About 0.05 means the GPU waits, on small batches, on overhead, or on a starved input pipeline. The numerator counts matmuls, so every second spent elsewhere lands as a lower MFU.

Fix the batch size and the input pipeline before you ask for more cards.

Peak is a ceiling under ideal conditions. GPU specs list it by dtype, so the denominator moves with the precision you run. An H100 lists a much higher peak for fp16, bf16, and fp8 tensor cores than for float32. Dense models often reach about half of it. Eight H100s running for one week do about 10²¹ FLOPs, comparable to major training budgets.

Precision is a placement decision

Store the parameters and the optimizer state in float32. Run most forward and backward matmuls in bfloat16. Keep the sensitive operations in float32, and those are often the attention-related ones.

float32 spends 32 bits as one sign bit, eight exponent bits, and 23 fraction bits. Exponent bits carry range and fraction bits carry precision. Every cheaper dtype is a decision about which of the two to give up.

The master copy stays in float32 because small updates vanish when the fraction bits run out.

fp8 goes further, with variants that trade range against precision again. H100 tensor cores support it, and it buys back speed and memory. Training on it safely is harder, so it lives inside careful mixed-precision configurations.

Mixed precision buys less memory for activations and intermediates and more throughput on tensor cores. It charges instability when the precision drops too low in the wrong place, and it often needs loss scaling. Training usually needs float32 somewhere to stay stable. Inference can quantize harder later, down to int4 in some deployments.

Memory is spent four ways, and views are free

Memory goes to parameters, activations, gradients, and optimizer state. Parameters are the learnable weights. Activations are the intermediate values saved for the backward pass. Gradients match the parameter shapes. Optimizer state is the extra buffers, and its size depends on the optimizer.

Tensors hold all four, plus the data itself as token IDs and embeddings. Take a deep linear model with hidden size D and L layers.

• Parameters are about L × D², plus a small head.
• Activations are about batch_size × sequence_length × D × L.
• Gradients equal the parameter count.
• Optimizer state is one times parameters for AdaGrad-style methods and two times for Adam-style.

Total bytes are those four added, multiplied by the bytes for each value. Three of the four scale with the parameter count. Activations scale with batch size and sequence length instead, which is the term that pulls the 40B answer down.

Tensor memory is elements times bytes for each element. A 4 × 8 float32 tensor holds 32 elements at four bytes, which is 128 bytes. Weight matrices reach gigabytes on the same arithmetic.

A tensor is a view into storage. Storage is a flat one-dimensional array of values. The metadata holds shape and strides, and the strides say how far to step through storage along each dimension.

Slices, transposes, and views share one storage and copy nothing. Mutate one view and every tensor on that storage changes with it. Transpose and some slicing produce non-contiguous views, and a call to contiguous() copies. Views are free. Elementwise operations and some reshapes allocate.

PyTorch creates tensors on the CPU by default, and a large model trains too slowly there. Moving data between CPU RAM and GPU HBM is expensive, so avoid the transfers you do not need. Know which device every tensor is on, and use tools that report it when you debug.

The Builder Test

Pick a run you have not started. Write both estimates down before you launch, the operations and the bytes. Then launch it, measure step time and peak memory, and compare against what you wrote.

When the estimate misses, name the term you left out before you change any code. Read peak memory against the four buckets and find which one you sized wrong. Check the activation memory, the dtype the matmuls ran in, and the MFU your input pipeline supports.

A sketch that is wrong on paper costs minutes. The same error found on 1,024 cards costs the run.

What Carries

You must be able to explain a run before you start it. Compute answers whether it finishes. Memory answers whether it starts.

A compute miss costs days. A memory miss ends the job before the first step.

The overview note called the pipeline a chain where every stage prices the next one. This is the stage where the price becomes a number you can check.

The bill is priced. The next choice is the shape of the model that spends it. Pre-norm, RMSNorm, SwiGLU, and RoPE get decided there. Settle these two numbers before you touch that design.

Reports put GPT-4 near 1.8 trillion parameters and its training cost near 100 million dollars. At that price, efficiency is the difference between a run you can afford and one you cannot. The tokenizer is the first stage where it is won or spent, and it is settled before training starts.

Scale matters, and efficiency decides how much scale you can afford

Training clusters reach about 200,000 H100 GPUs. At those prices you cannot waste compute. Frontier labs are usually more careful about it than academic runs are.

Most people quote the bitter lesson as proof that only scale matters. I read it as a function with two terms.

You buy accuracy two ways. Spend more resources on the same algorithms, or improve efficiency and get more from the resources you hold. ImageNet training between 2012 and 2019 gained about 44x in algorithmic efficiency at the same accuracy. Without those gains, that accuracy costs about 44 times more compute.

Most of the components predate GPT-3. Shannon used language models to estimate the entropy of English. Classical NLP put them inside machine translation and speech recognition. Attention, Adam, and the transformer arrived through the 2010s.

ELMo and BERT showed that one base model fine-tunes to many tasks. Researchers ran early 100B-parameter experiments with mixture of experts and multi-GPU parallelism. Frontier labs supplied the engineering and the decision to spend large amounts of money. Scaling laws became the design principle over the top. The ideas sat on the shelf. Efficiency and money moved them.

The field already ran the resources-only experiment. By 2007 Google trained 5-gram models on about two trillion tokens, more text than GPT-3 saw. Those models stayed shallow, and none of the behavior people now expect from a language model appeared. Resources bought better word statistics and stopped there.

Efficiency is the theme under every choice here, because compute is scarce today and text is abundant. Filter the corpus so that compute does not go to junk tokens. Shorten sequences so that the same budget covers more text. Train one epoch over a huge dataset, since more unique data once beats the same data twice.

The same pressure sets the defaults further down the chain. Architecture choices survive because they train and serve efficiently. A well-tuned configuration beats a naive one by an order of magnitude on the same hardware. Fine-tuning lets a smaller base model handle a narrow task, which cuts the serving bill.

That regime can flip. When compute grows faster than high-quality data, one pass over the full dataset stops being optimal. Repeated passes or careful sampling matter more. Architectures shaped by compute efficiency change when data becomes the bottleneck. Name the factor that limits you, then let the choices follow from it.

Small experiments hide two things that only appear in large runs

Frontier labs keep the training data, the architecture, and the procedure private. They name competition and safety as the reasons. So the money and the information sit inside a few buildings, and the rest of us learn on small models. Small models hide two effects that decide what happens in a large run.

The cost profile moves first. In small transformers, attention layers and MLP layers carry similar FLOPs. Near 175B parameters, MLP FLOPs are the largest part of the bill. Optimize attention on a small model and you can miss the cost that dominates at the top.

Threshold behavior hides the second one. Some tasks show no gain as training FLOPs rise, then improve quickly past a point. In-context learning is one of them. A small run reports that the approach fails, and the report is true only at the size you ran. So write the scale beside every negative result you carry out of a small experiment.

A tokenizer is a budget decision before it is a text decision

Efficiency is a chain of decisions, and the pipeline note put tokenization at the head of it. Every token the tokenizer emits is compute that somebody pays for later. The stages after it inherit whatever it decides.

A tokenizer maps raw strings to sequences of integer tokens and back. Reversibility is the hard requirement, because decoding has to return the string you started with. What it decides after that is budget: sequence length, vocabulary size, training cost, and which scripts get represented cheaply.

The model reads integer IDs. The tokenizer decides which chunks get an ID of their own. Everything else gets spelled out piece by piece. That vocabulary is the alphabet the model gets, and it never sees the text any other way.

Compression ratio is the number that turns a text decision into a compute bill. Divide bytes by tokens. GPT-2 on sample text runs about 1.6 bytes per token. Change the tokenizer and that ratio moves, and the length of every sequence you ever train on moves with it.

The same ratio prices serving. Prefill reads the whole prompt at once and stays compute bound. Decode emits one token at a time for every user and stays latency sensitive. The total serving cost across all users can pass the training cost. The tokenizer sets the token count on both bills.

Look at what a GPT-style tokenizer produces. Tokens are word pieces rather than whole words. Spaces ride inside tokens, so " hello" and "hello" are two different tokens. Numbers break into pieces that do not match the digit groups people use.

Character, byte, and word tokenizers each break in a different place

Each naive scheme fails somewhere different, and the failure tells you what a learned tokenizer has to fix.

Character tokenizers treat each Unicode character as one token. The scheme is simple and reversible. The vocabulary runs very large and fills with rare characters. Compression stays mediocre, and some characters take more than one byte anyway.

Byte tokenizers convert to UTF-8 and treat each byte as a token. The vocabulary holds 256 entries, and nothing falls outside it. Compression is one byte per token, so sequences run long and the attention cost climbs quickly. Byte-only models stay off the frontier for that reason, elegant and too expensive to run.

Word tokenizers split with a regex into words and non-word spans. Frequent words become single tokens and compression is strong. The vocabulary grows without limit, many words appear exactly once, and any new word arrives out of vocabulary. Training and evaluation both get harder.

Every naive scheme lands in one of three states: too big, too slow, or brittle. A tokenizer learned from data is the way out of all three.

Byte Pair Encoding learns its merges from the data

BPE started in 1994 as a compression method, and tokenization runs the same loop. Start from bytes rather than words. Merge frequent adjacent pairs into new tokens, and learn which pairs those are from the corpus you will encode.

Start with a byte sequence, values 0 to 255. Keep an empty merge list and a vocabulary that maps token IDs to byte strings. Then repeat for a fixed number of merges.

• Count how often every adjacent token pair appears in the data.
• Find the most frequent pair.
• Create a new token ID for that pair.
• Record the merge rule from the left and right tokens to the new ID.
• Replace every occurrence of the pair with the new token.

Each merge adds one vocabulary entry and shortens the sequence wherever that pattern is common.

Frequent patterns collapse into single tokens, and sequences get shorter. Rare patterns stay composed of smaller tokens, so nothing arrives unknown. You keep the compression of a word tokenizer and the coverage of a byte tokenizer.

Encoding converts the string to bytes, then applies the merge rules in the learned order. Decoding maps the IDs back to byte strings, joins them, and decodes to Unicode. More merges buy more compression, because more common sequences collapse to a single token.

The merge list is what you ship. You learn it once and apply it in that exact order at encode time. Apply the rules out of order and the same string tokenizes differently. It stays fixed for the life of the model. The text in the merge corpus is the text it compresses well later.

Real implementations pre-tokenize first. Split the text with a word-like regex, then run the merges inside each segment. Skip that step and merges run across whitespace, which gives tokens that straddle word boundaries. Pre-tokenization strongly controls how the finished tokens behave, and it is where spaces and punctuation get their rules.

A naive implementation is easy to read and slow, because it rescans whole sequences across every merge. Production tokenizers look only at the merges the current sequence allows. Their data structures avoid the full rescan. They also carry special tokens for start, end, and padding.

Vocabulary size buys sequence length and charges you elsewhere

Every merge you add buys shorter sequences and charges you somewhere else. A larger vocabulary packs more text into fewer tokens. It also grows the embedding table and leaves each rare token with fewer examples to learn from.

The charge lands unevenly across text. A common English word costs one token. A rare name, a typo, a code string, or a language the merges never saw costs many tokens. Each of those tokens spends context and compute that the common word did not.

Script coverage is decided in the same place. A vocabulary learned mostly on English gives other scripts few merges, so those strings fall back toward bytes. The same sentence then costs more tokens on every request.

The Builder Test

Calling an API teaches you the surface. Running text through your own tokenizer teaches you the bill. Take these strings and read the token counts.

• A few names, including some outside the Latin alphabet.
• A long URL from a page you use.
• A block of code from the repository you work in.
• A line of math with symbols and digits mixed together.
• A paragraph in a language other than English.

Compare each count against the bytes per token you measured on ordinary prose. Whatever comes back long is what the model finds expensive. Fix it in the tokenizer, or pay for it in every run after.

Then run the same strings through a second tokenizer and read the two counts side by side. The gap between them is what the choice is worth. Keep that file, and every tokenizer you consider later gets measured against it.

What Carries

Tokens are the interface between language and computation. A rough interface shows up as wasted context and as text the model cannot copy back cleanly. Neither symptom looks like a tokenizer problem when you meet it.

Byte-level models can remove tokenization one day, and none of them leads at the frontier now. So the tokenizer stays a decision you own and defend with a number.

The tokenizer fixes how much compute each string costs. The next thing to price is one training step, before you run it.

A language model is a pipeline of choices that turns text, compute, and feedback into behavior. Every stage sets the terms for the stage after it. Most LLM tutorials teach you to call an API. This sequence builds the system behind it.

Collect the data, tokenize it, build a transformer, train it, evaluate it, and serve it. Every hard part gets an answer you can check. By the end you can name what every number in a result cost to produce.

Each stage decides what the next stage can do

The chain runs from data collection through tokenization, architecture, optimization, hardware, scaling, evaluation, inference, and post-training. Each stage carries one assignment.

• Data decides what signal exists.
• Tokens decide how text becomes compute.
• Architecture decides what can be represented.
• Training decides what the weights record.
• Evaluation and serving decide whether anyone can use the system.

Tokenization is the first stage, and the token count it produces is what a training run pays for. Resource accounting belongs to training, where it answers whether a planned run fits the hardware. The training loop decides whether that run stays stable. Evaluation runs last, beside serving, and it gates whether the work leaves your machine.

A rough tokenizer makes training harder. Careless data makes the loss dishonest, and a loose evaluation makes a weak model look ready. Each failure lands one stage after the choice that caused it.

Small models teach the full process

You will not train GPT-4 here. The models are small, and the process is the same. The choices that are important at 7B parameters are the choices that are important at 70B.

The numbers shrink. The work stays the same: measure, debug, account for what you spent, and know what changed. A small model becomes a toy when you skip a stage.

A small run gives you the whole chain at a price you can pay many times. Run it end to end, break it, and read what the break cost.

Going straight to architecture is the expensive mistake

Architecture is important, and it is still one link in the chain. A beautiful model trained on careless data is a careless system. No block design recovers signal the data never carried.

When a result disappoints, walk the chain backwards from the number: evaluation, then training, then tokens, then data. Stop at the first stage you cannot explain, and fix that one. Rebuilding the later link is the expensive way to learn that the earlier link was the problem.

See the whole chain once, then go back to the hard parts

Read the chain once at speed. A pass that stops to master every step never finishes. The first pass buys the shape and nothing else, and after it you know which parts are hard.

Then return to the parts that were difficult. If a part stays difficult, slow down and stay on it until you can explain it in your own words. A part you can only repeat in borrowed words is a part you do not have yet. The goal is to build it again without the notes.

The Builder Test

Describe the path from raw text to one generated token. Do not omit a step. Mark every step that sounds like magic, and study those first. The description is finished when every step names what it decides.

What Carries

The pipeline is the object you are building. When something breaks, find the break before you reach for a bigger model. The bigger model hides that break and charges you on every run after. The chain starts at tokenization, and that is a budget decision before it is a text decision.