Parallelism Is the Hidden Curriculum

If communication dominates compute, an expensive cluster waits instead of learning. An idle accelerator costs the same per hour as a busy one. The scarce resource picks the parallel strategy.

Frontier Training Is Distributed Systems Engineering

The syllabus page behind this note is mostly a reading list. Read the references and one lesson repeats. Frontier training of a language model is distributed systems engineering.

Large models exceed the memory and throughput of one accelerator. You slice the training job across devices because nothing else fits. The split is mandatory. The communication bill arrives with it.

You do not need a frontier run to meet this constraint. Any job that outgrows one device inherits the same accounting.

Three Splits, Three Bills

The hard part is deciding what each device stores, what each device computes, and when the devices talk. Three splits answer that, and each one moves the pressure to a different place: memory, communication, synchronization, or idle time.

Data parallelism replicates. Each worker takes a different batch and holds a full copy of the model and the optimizer state. The bill is replication in memory plus a gradient collective on every step. It is the simplest split to write and the first to run out of memory.

Tensor parallelism splits the large matrix operations across devices. A giant layer no longer has to fit on one device. It pays in traffic inside the layer, on every forward and backward pass.

Pipeline parallelism assigns layer ranges to stages and streams microbatches through them. A stage at the front finishes early and waits for the stages behind it. That wait is the bubble. Microbatches shrink the bubble and never delete it.

Real systems combine all three. Sharding saves memory because it makes ownership explicit. That explicit ownership is what you audit when the run stalls.

Communication Is Part Of The Computation

Every split creates messages. Activations, gradients, parameters, and optimizer states move between devices while the arithmetic waits for them. The hidden curriculum is communication.

Nothing is free once it has to move. Parameters, activations, gradients, optimizer state, and tokens compete for the same memory and the same bandwidth. A blocking collective sits inside the step time, and the profiler charges it to the step.

Topology decides which of those messages is cheap. A split that ignores the wiring pays for every message at the slowest link.

Pipeline bubbles, slow synchronization, uneven shards, and memory imbalance make a large cluster behave like a much smaller one. Measure the ratio before you buy another rack. If message time per step grows faster than arithmetic time, more hardware makes the ratio worse.

Name The Wall Before You Choose The Tool

Bottlenecks move. You start short on memory, then short on bandwidth, then short on utilization, then short on patience for debugging.

Upstream choices move them. Width and context length decide how much memory one step needs, and the modeling note filed both under representation. A tokenizer that cuts text into smaller pieces spends more positions, and more positions cost memory. The token budget decides how many steps the run must survive, and the scaling-laws note picks that budget.

Activation checkpointing is the worked example of trading one wall for another. You drop the stored activations and recompute them during the backward pass. Memory pressure falls. Compute per step rises. Take the trade when memory is the wall, and refuse it when compute is.

A Week-Long Run Is A Reliability Problem

A week-long run is a reliability problem before it is a training problem. Throughput, device utilization, stragglers, failed workers, synchronization stalls, and numerical instability are first-class signals. Debugging a distributed job needs observability at the cluster level.

One lost worker costs every step since the last checkpoint. Every other device recomputes that stretch, so the whole cluster pays for one failed machine. The checkpoint interval is a budget decision, and the recovery path belongs in the plan.

A straggler never crashes, so nothing raises an alarm. The step ends when the slowest device finishes, and one slow shard sets the pace for the whole run. Numerical instability belongs on that dashboard too. A diverged run wastes as many device-hours as a dead worker.

The Builder Test

Draw the data movement for one training plan. Say where parameters, activations, gradients, and optimizer state live at every step. Mark what is replicated, what is sharded, what is communicated, and when devices wait.

Then hold the drawing against a real trace. Collectives, bubbles, synchronization, imbalance, memory pressure, and topology all appear there. A plan you cannot draw is not a plan yet. The trace names the part you guessed.

Measure the devices doing useful work at the same instant. That count is smaller than the rack count, and it is the one that predicts the finish time.

What Carries

Scaling training means keeping useful work in motion while communication and memory stay under control. Large-model parallelism is resource ownership plus communication scheduling. Once you can keep a cluster busy, the next question is what to buy with it.