Transformers Are Controlled Editing Machines

Show me the next open model and I will guess its architecture before I read the paper. Decoder-only, pre-norm, RMSNorm, SwiGLU, RoPE, no bias terms, and a width-to-depth ratio from a narrow band. The guess lands, because those choices train stably and serve cheaply.

You can guess a new open model's architecture and be right

Many labs trained many models. The designs that come back run after run are the ones that survived measurement. Copying them is the cheapest thing a builder does, and knowing why they won is the part that pays.

Two bills drive the convergence. A design that spikes during a long run costs you the run. A design that drags a large KV cache costs you every request you serve.

Four defaults moved away from the original transformer. Post-norm became pre-norm, and additive position embeddings became RoPE. ReLU inside the MLP became SwiGLU, and most linear layers lost their bias terms.

The rest did not move. Token and position information enters a stack of blocks, each carrying attention, an MLP, residual connections, and normalization. A serial block runs attention and then the MLP, each part with its own pre-norm and its own residual add. A final softmax over the vocabulary gives next-token probabilities.

The hyperparameters cluster the same way. Families keep d_model equal to n_heads times d_head, which keeps the attention sizes predictable. Too many heads leave d_head small, and a small d_head limits expressiveness. The pretraining loss follows the total parameter count more than the depth, so the shape sits in a broad band. At fixed compute, more depth helps some downstream behavior.

Vocabulary size shifted between generations. English-focused models used 30k to 50k tokens, and newer multilingual and production models use 100k to 250k. The larger vocabulary shortens sequences across scripts. That is the tokenizer note's compression ratio, arriving here as an architecture number.

Pre-norm keeps the residual path close to identity

The original transformer ran a sub-block, added the residual, then normalized. Pre-norm reverses that order. It normalizes the input to the sub-block, then adds the sub-block output to the residual stream.

The order decides what the residual path carries. Under pre-norm it stays near identity, and gradients flow more cleanly through deep stacks. Post-norm still trains, and it needs more careful warmup and more careful tuning to avoid loss spikes. Check the warmup first when a deep stack spikes early.

Some models add a normalization after the sub-block and leave the residual stream itself un-normalized. That double-norm pattern is a stability tool in the largest runs.

RMSNorm drops the mean and the bias and loses nothing

LayerNorm normalizes by the mean and the variance, and it carries both a scale term and a bias term. RMSNorm normalizes by the root mean square and keeps a learned scale only. Dropping the mean subtraction and the bias costs no quality. RMSNorm is cheaper and faster, which is why it dominates recent large models.

The same reasoning strips bias terms out of most linear layers. The parameter count falls, the kernels get simpler, and stability improves in many large runs. Bias terms add a small amount of expressive power, and that gain does not pay for its cost. Those edits are cheap, and together with pre-norm they buy stability and runtime.

Gated MLPs won on repeated measurement

Early transformers used ReLU, and GPT-style models popularized GeLU. Across many training runs, GLU variants reached lower loss and better downstream results at similar parameter counts. SwiGLU is the strongest of them, and gated MLPs are the default now.

The gated MLP splits the input path in two.

• Compute a main projection and a gate projection.
• Apply a nonlinearity such as GeLU or Swish to the main path.
• Multiply the main path and the gate path elementwise.
• Project the result back to the model dimension.

GeGLU and SwiGLU are the common variants, and SwiGLU uses Swish for the gate. The gate works as a learned filter over the hidden dimensions, deciding which channels survive the multiply.

The extra projection costs parameters, so the hidden width comes down to pay for it. A non-gated MLP uses d_ff = 4 × d_model. A GLU MLP uses d_ff near (8/3) × d_model, about 2.66 times the width. Three projections at 2.66 times the width cost what two projections at 4 times the width cost. The parameter count holds, so the comparison between the two MLP types stays honest.

RoPE puts relative position inside the dot product

Position methods varied, and they include sinusoidal absolute embeddings, learned absolute embeddings, relative-bias methods, and ALiBi. Those methods add a position vector at the bottom of the network. RoPE became the default for most dense models, and it never touches the bottom of the network.

RoPE leaves that formula alone and turns its inputs. It pairs up dimensions of the query and key vectors and rotates each pair by a position-dependent angle. The frequency differs for each pair. The dot product between a rotated query and a rotated key then carries the relative position between them. That placement makes RoPE compatible with context extension methods.

Stability tricks exist because large runs break without them

Gradient spikes and numerical problems around softmax end large training runs. The softmax exponentiates its inputs, so one extreme logit is enough to overflow. Two softmax sites carry most of that risk: the final vocabulary softmax and the attention softmax inside each block.

z-loss covers the output side. Add a small penalty on (log Z)², where Z is the softmax normalizer. The penalty discourages extreme logits and keeps the softmax in a safer numeric range.

QK norm covers the attention side. Normalize the queries and the keys before the dot product, and the attention logit scale stays under control. The overflow risk falls, and a larger learning rate sometimes becomes usable. Direct soft caps on the attention logits exist, and their results are not consistent.

One knob from the regularization era survives, and its job changed. The overfitting intuition does not transfer to pretraining. The data is very large and the model sees each example about one time. Dropout left most large runs.

Weight decay stayed because it helps optimization. It interacts with the learning rate schedule and improves the final training loss. The gain arrives late, after the learning rate decays.

Inference pressure explains MQA, GQA, and hybrid attention

Training is compute-heavy and runs in parallel across the whole sequence. Decoding produces one token at a time, so memory traffic dominates attention. The KV cache stores the past keys and values, so the model does not compute them again at every step.

The cache trades compute for memory traffic. Every step reads the whole cache back, and the cache grows with the context. Decoding becomes more memory-bound the longer the context runs.

MQA shares one set of keys and values across all query heads, which shrinks the cache by a large factor. GQA groups the heads so several query heads share one K/V set. GQA holds quality in most cases and still cuts the memory, which puts it between MQA and full multi-head attention.

Full attention is quadratic in the sequence length, so a very long context needs structure. Most layers now run local sliding-window attention with RoPE. A few layers run global full attention, and those global layers often drop RoPE.

Local layers handle nearby structure cheaply. Global layers mix information across the whole sequence, and only a few layers pay the quadratic cost. Removing the positions in those global layers can improve extrapolation.

Each choice here answers the serving bill, and the serving note is where that bill gets priced.

The Builder Test

Predict what a knob changes before you turn it. Depth, width, context length, and learning rate each need a stated reason and a stated expected effect. Write the prediction down before the run and compare it to what the loss did.

Change one knob at a time. A better loss then has a named cause, and a broken run has one thing to undo.

What Carries

A transformer block edits a stream of token vectors. Attention moves information between positions, and the MLP transforms each position. Normalization and the residual path keep the edits stackable. Every swapped default protects that stacking, during the run or during serving.

The model is the block plus the training recipe plus the data plus the compute budget. Architecture becomes behavior only through training.

The block is settled, so the next lever on capacity is how many parameters run for each token.