Scaling Laws Are Planning Tools

Chinchilla put twenty tokens on each parameter, and every team that measured the ratio again landed higher. Meta measured about 39 for LLaMA 3. MiniCPM measured about 192. The Hunyuan analysis measured about 96, counting active parameters rather than total ones.

Twenty tokens for each parameter is a floor

Chinchilla's ideas survive contact with production. Its constants do not. Architecture, data quality, and optimization decide where the ratio lands, and all three changed after Chinchilla.

Every one of those numbers came out of an isoFLOP sweep, the method the scaling-law note laid out. Fix the compute budget, vary parameters against tokens, fit the curve, and take the minimum. The constant that falls out belongs to that setup and to no other.

MiniCPM trained models from 1.2B to 2.4B parameters on far more compute than that size usually gets. The result is a small model that is strong for its size. Hunyuan is a mixture of experts, so its sweep counts active parameters. The method holds for that architecture and the constant it returns is specific to it.

The ratio climbed because of the bill the inference note priced. Serving cost tracks parameter count, and it repeats on every request the model answers. Training happens once. A smaller model trained on more tokens moves spend off the recurring side.

After Chinchilla and ChatGPT, most large labs stopped publishing their scaling recipes. Cerebras-GPT, MiniCPM, and DeepSeek kept publishing theirs. Their write-ups are where the full recipe still appears, so the working practice comes from them.

muP keeps the best learning rate fixed as the model widens

Scaling asks for three things. The isoFLOP sweep buys two of them, the parameter and token split and a predictable loss. muP buys the third, which is hyperparameters that survive a change of size. It holds the best learning rate steady as the width grows, and it does nothing else.

Cerebras-GPT trained from 0.1B to 13B parameters on Chinchilla-style compute-optimal budgets, and put standard parameterization next to muP. On a log-log plot of test loss against compute, standard parameterization wanders around the predicted line. muP tracks the line smoothly.

The learning rate is the cause. Under standard parameterization the best learning rate moves as the model widens. One global value then runs too large at the top end. Tune at a small width and the same rate breaks the wide model.

muP holds two things steady as the width grows. At initialization each activation coordinate stays order-one, so a layer's L2 norm grows like the square root of width. After one optimizer step, the change in each activation coordinate stays order-one as well. Those two conditions define width-invariant training.

Two rules follow. Initialize the weights at variance about one over fan-in. That is Xavier and Kaiming practice, and most code already does it.

Scale each layer's AdamW learning rate the same way. When fan-in is the model width, the layer's rate falls like one over width.

Standard parameterization already initializes close to what muP needs. The learning rate is where the two part.

Embeddings get a fixed scale of their own, and attention carries one more subtlety. Standard attention divides the logits by the square root of d. Some muP treatments divide by d instead, for tighter update stability.

MiniCPM ran the test. They swept learning rates at several model sizes and found a wide good region. The minimum sat in nearly the same place at every size. Their best settings matched Cerebras to about a constant factor, with the residual blocks scaled by depth.

That is why the search moves down in size. A large hyperparameter search runs on tiny models, and the wide model inherits the answer.

MiniCPM searched the aspect ratio, the learning rate, and the core settings on 9M and 30M models. Then it froze those choices and widened. The aspect ratio holds on the way up, so d_model, d_mlp, and head count keep their proportions.

muP fails in named places

Three changes break the learning-rate transfer.

• Extra learnable gains in the wrong places break the width invariance.
• Sign-based optimizers such as Lion do not match the assumptions muP is derived from.
• Strong weight decay can break it too.

Two changes leave it alone. Swapping the MLP nonlinearity keeps the transfer, and changing the batch size keeps it too. Small differences in initialization matter less than the learning-rate scaling.

Large studies tune the learning rate at a small width and then train wider models at that same rate. Under muP the loss-against-learning-rate curves agree across all the widths. Under standard parameterization the borrowed rate runs too hot as the width climbs.

Check the optimizer and the weight decay before you trust a transferred rate. One run reached about 10B parameters and trained successfully. Its learning rate came from small-scale muP sweeps, which is the entire return on the method.

WSD makes scaling experiments cheap

Chinchilla-style analysis needs many runs, several model sizes crossed with several token budgets each. Cosine schedules make that grid expensive.

The learning-rate curve depends on the total number of steps. A checkpoint from a long run is not the end of a shorter one. Every point on the grid pays for its own run from scratch.

WSD splits the schedule in three.

• Warmup raises the learning rate to its full value.
• A long stable plateau holds it there.
• A fast decay drops it to a small value at the end.

The plateau does the work. Rewind to an earlier plateau checkpoint and run the same decay from there. The result is a finished schedule for a smaller token budget. One long run yields a row of shorter ones, and you retrain nothing.

MiniCPM ran its isoFLOP analysis on those rewinds, with the lower-envelope method and a two-variable fit. That is where its ratio came from. DeepSeek runs a WSD-like schedule for the same reason, which is reuse across data budgets.

DeepSeek skipped muP and fit the hyperparameters directly

DeepSeek does not use muP. They scale the learning rate and the batch size against compute, and they fit those laws themselves.

The loop starts small. Sweep learning rate and batch size over a grid at small model sizes, then record the minimum. The minimum comes out broad rather than sharp at each level. Repeat at higher compute budgets and keep the best setting from each one.

Then fit the two trends and read them at the target run. Batch size fits cleanly against compute. Learning rate comes out noisier, and it still fits well enough to use.

Their isoFLOP curves came out smooth. At each compute budget they fit a quadratic in tokens and took the optimum. The trained models matched the prediction, which buys fewer surprises at the size where a surprise costs the most.

muP derives the scaling. DeepSeek measures it. Both spend small-model compute to set the dials on the large run, and you pay one bill or the other.

Critical batch size grows as the target loss falls

MiniCPM also measured where a larger batch stops paying. Past that point, more batch buys no further drop in loss. For each model size they trained across a range of batch sizes and recorded the final loss.

The point moves with the target. Push toward a lower loss and the critical batch size grows, on a relation close to a power law. Name the target loss and the batch size follows from the curve.

The Builder Test

Pick one hyperparameter you did not measure yourself, such as the learning rate in your plan. Find the run that produced it. Name the architecture, the data, and the optimizer that run used.

Then say whether your setup matches that one. The common trap is to carry a curve past the setup that produced it. Data quality, optimizer changes, architecture changes, and evaluation leakage all move the constants.

If the answer is no, the number is the start of a sweep. The plan needs the sweep before it needs the number.

I treat a borrowed constant as a dated measurement and check what changed since the date. The check that settles it is one isoFLOP sweep on your architecture and your data.

What Carries

Borrowed constants expire. Each one was measured under a setup, and your setup differs.

The isoFLOP loop replaces them with a number you own. muP and WSD exist to make that loop cheap enough to run again when the architecture or the data moves.

The run is planned and scheduled. The next question is how you will know whether the result is any good.