Inference Is Cache, Latency, and Scheduling

Generation is memory-bound. Every sequence you serve drags its own KV cache. Moving that cache is most of what the GPU does between tokens.

Training is expensive and rare. Inference runs continuously. At the volumes a company serves, a small gain per generated token beats a large gain on the training bill.

Chat is the visible use. Evaluation, code completion, batch processing, test-time reasoning, and RL sampling all run the same loop. The serving path carries research work as well as product traffic.

The scaling note priced the training run and left serving out of the budget. Serving is the bill that keeps arriving. Three levers move it: shrink the KV cache, share it, or spend compute to skip decode steps. Everything after that is scheduling.

Three numbers describe a serving system, and they do not move together.

• TTFT is the time from prompt arrival to the first output token. It is mostly prefill, and users notice it first.
• Per-token latency is the gap between output tokens after the first. Users feel it as smoothness.
• Throughput is total tokens per second across all requests. Batch jobs and high-volume traffic optimize this one.

Decode is memory-bound and prefill is not

Arithmetic intensity says which resource runs out first.

High intensity puts the ALUs at the bottleneck. Low intensity puts memory bandwidth there. On an H100 the crossover sits near 295 FLOPs per byte.

Take one matmul. X is B × D and W is D × F. The work costs about 2 × B × D × F FLOPs. In a language model D and F run much larger than B, so the intensity comes out at about B. At B = 1 the matmul becomes a matrix-vector product. Intensity falls to about 1 FLOP per byte, against a crossover of 295.

The note that priced a training run counted FLOPs against a budget. Serving counts bytes moved per token. The bandwidth note showed compute growing faster than memory bandwidth. Decode is where you pay that gap.

In training you hold the full sequence. You parallelize across sequence length and run matmuls large enough to saturate the GPU. Decode cannot borrow that trick, because token t+1 waits on token t. The GPU stays underused unless you raise the batch size or run many sequences.

Prefill and decode split on this axis. Prefill runs the whole prompt at once, parallel over sequence length, with high intensity and compute to spare. Decode adds one token at a time, conditions on the full history, and reads far more than it computes. Prefill is the efficient half. Decode is the bottleneck a production server meets.

The KV cache turns quadratic work into linear traffic

Without a cache, every new token re-attends over all previous tokens. Generating T tokens costs O(T²) work for one sequence. That is too slow to serve.

The cache holds the keys and values of every past token, for every layer and every head. A new token computes its own Q, K, and V, then attends over the cached past. Cost per token drops to O(T).

You pay for that saving in traffic. At every step attention reads large cached tensors out of HBM and does little math on them. That ratio is what holds attention on the memory-bound side.

The cache is also the serving bill. Longer prompts, more users, and larger batches compete for the same memory. The weights are the fixed part of that budget.

A larger batch helps the MLP and does nothing for attention

The MLP during generation has intensity of about B. Raise the batch and the MLP walks toward the compute-limited side, where a GPU earns its price.

Attention does not move. The sequences share one set of weights, and each one carries its own KV cache. KV traffic grows with B, so B cancels and the intensity stays near 1 at every batch size.

LLaMA 2 13B on an H100 prices the dial.

• At B = 1, per-token latency is about 8 ms and throughput passes 100 tokens per second.
• At B = 16, the KV cache grows 16 times. Per-step latency rises, and throughput rises with it.
• Past that point, latency keeps climbing while the throughput returns shrink.
• At B = 256, the cache can pass 80 GB, which is more than the GPU holds.

A small batch buys low latency and low throughput. A large batch buys high throughput, higher latency, and memory pressure.

Scaling out is the easy half. Run M copies on M GPUs, and per-request latency stays about the same while total throughput scales with M. Decode needs no cross-GPU synchronization per token, which is what keeps the copies independent.

Shrink the cache or share it

KV movement dominates decode, and four architecture choices attack that traffic directly. GQA arrived in the architecture note and MLA in the mixture-of-experts note. Both were chosen against this bill.

• GQA cuts the KV heads from N to K and lets several query heads share one. The cache shrinks by about N/K, which is why LLaMA 3 ships it. Push K too low and accuracy drops.
• MLA projects K and V into a latent space before caching them. DeepSeek V2 compresses a 16,000-dimension KV space to 512.
• CLA shares the KV projections across several layers, so those layers share one cache and one stream of traffic.
• Local attention keeps a sliding window of size K, so the cache grows with the window instead of the sequence. A pure local stack loses long-range dependencies. Designs answer that with mostly local layers and a full-attention layer every sixth layer.

Another line of work fixes the state size instead of trimming it. State-space models carry a fixed-size state and update it for each token. Each step costs linear time and constant memory. Early versions were weak at associative recall, and Mamba works well on language at modest scale.

Linear attention approximates softmax with kernel features and keeps a running state, again constant memory per step. Some large systems run mostly linear and local attention, with a few full-attention layers for quality.

Quantization is the other lever on bytes. The ladder runs FP32, BF16, FP8, INT8, INT4. Fewer bits per weight moves fewer bytes, and it helps the weights and the cache fit at all. Too much quantization costs accuracy, and some weights carry outliers that need higher precision.

AWQ calibrates to find the weights that matter, drops the rest to INT4, and reaches about a 3× speedup. A 15B model pruned to 8B, then distilled from the large model, keeps benchmark quality close. Both trade fidelity for speed, and both work while the quality loss stays small.

Speculative decoding buys about two times and changes nothing about the output

Checking tokens in parallel costs less than generating them one at a time. Speculative decoding spends that difference.

The target model Q is large, accurate, and expensive. The draft model P is small, cheap, and similar enough in behavior to guess what Q says next.

• P proposes K tokens autoregressively from the current context.
• Q scores all K tokens in one parallel pass.
• Accept each proposed token with the ratio Q(token)/P(token), clipped at 1.
• On a rejection, sample the corrected token from Q.

The output is an exact sample from Q. The distribution matches a run of Q alone, so quality is not part of the trade. When P is fast and stays close to Q, acceptance runs high. Speedups of about 2× or more are common.

Medusa drafts several branches at once. EAGLE guides the draft with signals from the target.

Real traffic is a scheduling problem

Production traffic is irregular. Requests arrive at unpredictable times, and prompts and generations differ in length. Many requests share a prefix.

Dynamic batching runs one step for every active sequence. The scheduler drops finished requests and admits new ones, so the batch is rebuilt continuously.

Selective batching splits the step. Attention needs the KV history of each sequence, so mixed lengths stay awkward there. MLP work does not care, so the server flattens token vectors across sequences and runs shared matmuls.

Sequence lengths are unknown when a request arrives. One contiguous buffer per request wastes memory while the request is short, and leaves a hole when it finishes.

PagedAttention treats the KV cache the way an operating system treats memory. The cache splits into fixed-size pages, and each sequence holds a list of pages instead of one block. The server allocates pages wherever space is free and reclaims them when a sequence ends.

Shared prefixes get the same treatment. The server stores the shared pages once and counts references to them. When two sequences diverge, copy-on-write allocates new pages for the divergent part alone.

High throughput does not imply low latency. A server can push many tokens per second while some requests stall in the queue. Batch policy and scheduler decide who waits.

The Builder Test

Split every serving metric into prefill and decode before you change anything.

• Report TTFT and per-token latency separately. One blended latency number hides which half is slow.
• Measure the KV cache at your real batch size and context length. If it does not fit, the batch size is already decided for you.
• Name the side each optimization touches. A kernel that speeds up prefill does nothing for a decode-bound server.

If you cannot name the slow side, you are tuning by guess.

What Carries

Inference speed comes down to what you can reuse and what you can fit. The KV cache is both. Design as though memory is the constraint, and cut the bytes you move per useful FLOP.

Serving cost now sits in the run plan next to the training bill. The next note reads the measurements production teams published once they planned that way.