Scaling Laws Make Compute Legible

Small probe runs estimate the slope before a frontier run spends real money. The curve they return speaks for the setup that produced it and for nothing else. Carry it past that setup and the number it gives you is a guess.

Scaling Laws Turn Ambition Into Accounting

A scaling law measures how loss changes as you vary model size, dataset size, and compute. Kaplan-style work found smooth power-law improvement across all three. Smoothness makes the small experiment worth running, because a small run forecasts a large one. The goal is to put a training budget where it buys the most capability.

Train several smaller models across a range of sizes and token counts, then fit the trend. The fit turns a future expense into a present decision, and it makes the large run less blind.

I read a pilot as a purchase of information about the slope. A pilot that returns a leaderboard number and no slope buys nothing. Answer the resource-allocation question first, and chase the leaderboard after.

Compute-Optimal Training Balances Parameters Against Tokens

Compute-optimal work sharpened that lesson into an allocation rule. A fixed compute budget splits between parameters and tokens. Spend too much on parameters and too little on data, and the model underlearns. Spend too much data on a small model, and the model cannot use the signal. Find the balance before the bill gets large.

You pay for both failures out of the same fixed budget. Every parameter you add is a token you do not train on.

A token budget is also a step count on a cluster. Nothing is free once it has to move. The parallelism note priced what each step carries: gradients, activations, parameters, and optimizer state. Every extra token buys signal and pays communication.

Diagnose The Bottleneck Before You Buy The Next Parameter

Four diagnoses cover the common cases: parameter-limited, data-limited, compute-limited, and evaluation-limited. A bigger model is a strategy only when the data budget already fits. Otherwise the new parameters sit on top of a corpus that cannot feed them.

Each diagnosis names a different purchase. Parameter-limited buys capacity, and data-limited buys tokens or a cleaner corpus. Compute-limited shrinks the plan until it fits the budget you hold.

The spend has four axes. Fixed compute buys parameters, tokens, data quality, and evaluation. Cleaning the corpus and building the evaluation draw on the same budget as the run.

Evaluation-limited is the case a loss curve cannot see. Loss falls, and the number the product cares about stays flat. Downstream evidence settles it. The capabilities note supplies the form.

Record the interaction contract, then read the failure slice under it. Prompt format, decoding, and scoring all belong in that record. Fix the evaluation first, because a blind measurement leaves every other diagnosis unproven.

The Curve Speaks Only For Its Own Setup

The fit is empirical. Someone measured points and drew a line through them, which is why the measured range matters. Five assumptions sit behind that fit: data quality, model family, optimizer, token budget, and evaluation target. The line holds while all five hold.

Smoothness inside the measured range says nothing about the region past it. Check the fit quality, and check that the planned run lands inside the range you measured.

The data note gives the mechanism behind a data-quality shift. Mixture weights decide what the corpus treats as normal. Re-weight the mixture and the fit now describes a different object. Contamination bends the line from the other side, because the evaluation set leaks into training.

A changed optimizer or a changed architecture voids the fit the same way. The old slope then describes a model you no longer train.

Scaling Scales The Risks Too

The curve forecasts loss and stops there. Memorization, misuse capability, energy use, and deployment impact all grow with the run. Each one needs its own measurement on its own schedule.

Decide who owns each measurement before the cluster starts. The run produces all four whether or not anyone watches.

The same accounting continues after launch. A larger model costs more per token, on every call, for as long as the product runs. The marginal quality gain has to cover latency, money, and environmental cost. Price that gain against the traffic you serve, rather than against the benchmark you published.

The Builder Test

Take one scaling plot you rely on and name the decision it changes. The list is short: model size, token count, compute budget, or accepted risk. A curve that changes none of them is decoration.

Then write the setup beside the plot. Data mixture, model family, optimizer, token budget, evaluation target. If the planned run differs on any line, refit with a probe sweep before you spend.

What Carries

A scaling law is a planning instrument, and it holds for exactly as long as its assumptions hold. The forecast is conditional on a setup you own and must re-check.

A budget argument has a second answer. Stop spending the whole model on every token.