Training Is Math Plus Bookkeeping

A training step costs about six times the token count times the parameter count. AdamW costs about sixteen bytes for each parameter. Those two numbers price a run before you launch it.

The tokenizer note settled what a string costs in tokens, which makes it the first lever on this bill. The rest is arithmetic you can do on paper. I do it before I ask anyone for a cluster.

One step costs about six times tokens times parameters

Matmul dominates the cost of a training step. Count the matmuls and you have the whole estimate.

A B × D matrix times a D × K matrix produces B × K output elements. Each element does D multiplies and D adds. The total comes to about 2 × B × D × K.

The units matter when you read a spec sheet. One FLOP is a single floating point add or multiply. FLOPs is a total count of them. FLOP/s is a rate.

Take a two-layer model. X is B × D, W1 is D × D, and W2 is D × K. Forward computes H1 = X × W1, then H2 = H1 × W2. That first matmul costs 2 × B × D², and the second costs 2 × B × D × K. Together they come to about 2 × B × (D² + D × K).

D² + D × K is the parameter count of that model. B is the number of data points. So the forward pass costs about two times data points times parameters.

The backward pass runs two matmuls for every one the forward pass ran. One produces the gradient for the weights and one passes the gradient down to the input. For W2 that is H1 transposed times grad_H2, at about 2 × B × D × K. Then grad_H2 times W2 transposed, at the same cost. W1 needs the same pair over its D × D shape.

Backward totals about 4 × B × (D² + D × K), twice the forward pass. Add the two and one step costs about six times tokens times parameters.

Language models feed batch, sequence, and hidden shapes into batched matmuls. The same weights apply to every token in every sequence. Data points are tokens, which is why the formula reads in tokens and parameters.

AdamW costs about sixteen bytes per parameter

AdamW in mixed precision costs about sixteen bytes for each parameter. Two bytes hold the bf16 weights and two hold the gradients. Four hold the fp32 master copy. Four more hold the first moment, and four hold the second.

A 7B model is roughly 112 GB of state before a single activation. An H100 has 80 GB. One card cannot train that model without sharding or gradient accumulation.

That sixteen-byte bill is the reason sharding exists, and the parallelism notes spend themselves on paying it down.

The optimizer decides eight of the sixteen, four bytes for each moment. SGD keeps no extra state. Momentum, AdaGrad, and RMSProp each keep one extra tensor for each parameter. Adam keeps two moving averages, the first moment and the second.

Answer two questions before you launch

Two sketches decide whether a run is worth asking for.

How long does a 70B transformer take on 15T tokens with 1,024 H100s? Total operations come to 6 × parameters × tokens. Pick the peak FLOP/s for the dtype you run, then apply an MFU guess. 50 percent is reasonable for a well-tuned configuration.

Multiply peak by MFU, then by the GPU count, then by the seconds in a day. That gives operations for each day. Training days are the total divided by that. The MFU guess is the soft part of the sketch. Carry it as a range and check it on the first day of the run.

If the day count comes back too high, the formula names the moves. Cut tokens, cut parameters, add GPUs, or raise MFU. Two of those change the model you get, and two change only the bill.

What is the largest dense model that fits on 8 H100s under AdamW with no memory optimizations? Each card carries 80 GB of HBM. Divide the total by sixteen bytes for each parameter. The answer lands near 40B parameters. Activations, sequence length, and other overhead pull it down from there.

Run the memory sketch first. A run that does not fit never gets to be slow. Leave headroom for activations, because the sixteen bytes cover none of them. If the model overruns the cards, shard it, accumulate gradients, or make it smaller. Each of those three changes the time sketch too.

MFU tells you whether the GPU is working or waiting

MFU is the operations your model needs divided by the operations the card can do in that time.

• Count the model FLOPs by adding the matmuls at two times the product of their dimensions.
• Measure step time.
• Divide operations by time for the measured FLOP/s.
• Divide that by the peak FLOP/s of the dtype you run.

Then read the number. About 0.5 or more is strong for a real system. About 0.05 means the GPU waits, on small batches, on overhead, or on a starved input pipeline. The numerator counts matmuls, so every second spent elsewhere lands as a lower MFU.

Fix the batch size and the input pipeline before you ask for more cards.

Peak is a ceiling under ideal conditions. GPU specs list it by dtype, so the denominator moves with the precision you run. An H100 lists a much higher peak for fp16, bf16, and fp8 tensor cores than for float32. Dense models often reach about half of it. Eight H100s running for one week do about 10²¹ FLOPs, comparable to major training budgets.

Precision is a placement decision

Store the parameters and the optimizer state in float32. Run most forward and backward matmuls in bfloat16. Keep the sensitive operations in float32, and those are often the attention-related ones.

float32 spends 32 bits as one sign bit, eight exponent bits, and 23 fraction bits. Exponent bits carry range and fraction bits carry precision. Every cheaper dtype is a decision about which of the two to give up.

The master copy stays in float32 because small updates vanish when the fraction bits run out.

fp8 goes further, with variants that trade range against precision again. H100 tensor cores support it, and it buys back speed and memory. Training on it safely is harder, so it lives inside careful mixed-precision configurations.

Mixed precision buys less memory for activations and intermediates and more throughput on tensor cores. It charges instability when the precision drops too low in the wrong place, and it often needs loss scaling. Training usually needs float32 somewhere to stay stable. Inference can quantize harder later, down to int4 in some deployments.

Memory is spent four ways, and views are free

Memory goes to parameters, activations, gradients, and optimizer state. Parameters are the learnable weights. Activations are the intermediate values saved for the backward pass. Gradients match the parameter shapes. Optimizer state is the extra buffers, and its size depends on the optimizer.

Tensors hold all four, plus the data itself as token IDs and embeddings. Take a deep linear model with hidden size D and L layers.

• Parameters are about L × D², plus a small head.
• Activations are about batch_size × sequence_length × D × L.
• Gradients equal the parameter count.
• Optimizer state is one times parameters for AdaGrad-style methods and two times for Adam-style.

Total bytes are those four added, multiplied by the bytes for each value. Three of the four scale with the parameter count. Activations scale with batch size and sequence length instead, which is the term that pulls the 40B answer down.

Tensor memory is elements times bytes for each element. A 4 × 8 float32 tensor holds 32 elements at four bytes, which is 128 bytes. Weight matrices reach gigabytes on the same arithmetic.

A tensor is a view into storage. Storage is a flat one-dimensional array of values. The metadata holds shape and strides, and the strides say how far to step through storage along each dimension.

Slices, transposes, and views share one storage and copy nothing. Mutate one view and every tensor on that storage changes with it. Transpose and some slicing produce non-contiguous views, and a call to contiguous() copies. Views are free. Elementwise operations and some reshapes allocate.

PyTorch creates tensors on the CPU by default, and a large model trains too slowly there. Moving data between CPU RAM and GPU HBM is expensive, so avoid the transfers you do not need. Know which device every tensor is on, and use tools that report it when you debug.

The Builder Test

Pick a run you have not started. Write both estimates down before you launch, the operations and the bytes. Then launch it, measure step time and peak memory, and compare against what you wrote.

When the estimate misses, name the term you left out before you change any code. Read peak memory against the four buckets and find which one you sized wrong. Check the activation memory, the dtype the matmuls ran in, and the MFU your input pipeline supports.

A sketch that is wrong on paper costs minutes. The same error found on 1,024 cards costs the run.

What Carries

You must be able to explain a run before you start it. Compute answers whether it finishes. Memory answers whether it starts.

A compute miss costs days. A memory miss ends the job before the first step.

The overview note called the pipeline a chain where every stage prices the next one. This is the stage where the price becomes a number you can check.

The bill is priced. The next choice is the shape of the model that spends it. Pre-norm, RMSNorm, SwiGLU, and RoPE get decided there. Settle these two numbers before you touch that design.