Post-Training Teaches the System What to Reward

Post-training decides how capability shows up in a product. Pre-training loads that capability into the weights and stops there. The same base model can come out helpful or harmful, depending on the steps that follow.

Every shortcut in the feedback becomes behavior. A rater with one minute for each pair rewards confident structure. The model trained on those labels learns to write confident structure. Nobody put that rule in the guidelines.

Pre-training packs capability and post-training decides when it appears

Pre-training runs over web text, books, and code. The model picks up language, reasoning patterns, coding, and many world facts. It does not come out of that behaving like an assistant. The model must obey instructions, stay helpful, stay safe, and work inside a product.

Supervised fine-tuning supplies the behavior. You collect pairs of instruction and good answer, then train the model to imitate the answer by gradient descent. This is supervised learning on expert demonstrations. Take a reasonable base model and public instruction data such as OpenAssistant or OpenHermes. Train with solid hyperparameters. You get a chat model that follows instructions and trails the best released ones.

Instruction data arrives from three sources. Each one charges a different price.

• Benchmark data converted to instruction format, the FLAN style, is large and cheap, and it teaches short multiple-choice answers.
• Human-written chat data, the OpenAssistant style, is slow and expensive, and it teaches what a real conversation looks like.
• Model-generated data, the Alpaca style, is cheap and consistent, and it teaches the quirks and hallucinations of the teacher.

Diversity is the price on the Alpaca style. A seed set of human instructions goes into a strong model such as InstructGPT. More instructions come back, and they stay short and similar to each other.

Safety tuning exists because people misuse models for scams, misinformation, and harm. A product also needs the trust of its users and its advertisers. Safety data is instruction data where the correct answer is a refusal or a safe redirection. A few hundred well-built examples mixed into the instruction set change how the model refuses. The effect is larger on a strong base model.

Instruction data no longer waits for a final stage. Engineers mix instruction-like data into late pre-training while the learning rate decays. Stage one stays classic, on web, code, and books. Stage two adds higher-quality instruction data and keeps training, pulling in Wikipedia, QA, chats, code SFT, and StackExchange. A small final SFT pass on the best instruction set is optional.

Mid-training uses instruction data in bulk. Keeping pre-training data in the mix cuts catastrophic forgetting. Each instruction example also returns more value. Many models shipped as base models already saw instruction-like data in mid-training. The clean split between base and chat is gone.

Fine-tuning on answers the model cannot support teaches invention

Quality in instruction data depends on what the model already knows. The same answer is good training for one model and a trap for another.

Take a training example that asks for an introduction on monopsony and ends with a specific citation. That answer teaches two things at once. One is a fact claim, that this reference exists and fits. The other is a shape, that a complex answer ends with a citation-like string.

A model that does not hold the reference learns only the shape. It then invents references that match the shape. The run taught it to copy the form of a good answer.

Post-training data more polished than the knowledge of the model teaches it to fake competence. Instruction tuning and RLHF both reward confident detail. Careless data teaches the model to always answer, even where a refusal is correct.

Good data sometimes says that it does not know. When the fact is missing, the correct target is a short admission or no answer at all. On-policy RL points the same way. It trains on the outputs of the model itself, and the updates land where the model already has some ability.

Post-training changes behavior more than knowledge. Mid-training adds a little, and a small SFT set with an RLHF run does not replace broad pre-training. The data notes put the ceiling in the corpus. Post-training decides how much of that ceiling a user ever sees.

RLHF changes the objective from matching a distribution to scoring well

Pre-training and SFT assume a target distribution of good completions. Next-token prediction pulls the model toward it. Success means the model looks like its training data.

RLHF drops the target distribution. You define a reward over the prompt and the answer. The model becomes a policy that picks answers scoring high on that reward. Success means a high score from a human or from a reward model.

A KL penalty holds the policy near a reference, usually the SFT model. Remove it and the policy drifts.

Two things make judgments worth the switch. A supervised example needs a full expert answer for every prompt. A preference label needs one choice between two candidates, and it costs less to produce. People also write worse answers than the ones they choose when shown options. That distance is the generator-validator gap, and RLHF runs on it.

The InstructGPT-style pipeline is short. Start from an SFT model and sample several answers for each prompt. Show annotators the pairs and ask which answer is better, with a tie or both bad allowed. Fit a reward model to those choices. Run an RL algorithm that raises the score of the policy.

A rater with one minute cannot check facts

The guidelines ask for helpful, truthful, and harmless answers. The rater also judges style, since the answer must stay polite and clear. Toxic content and hallucinations are supposed to lose.

The budget is about one minute for each example. Correctness is the part that does not fit, and math or fact-heavy answers are the worst case.

Checking one factual claim takes four steps.

• Understand the question that was asked.
• Read both answers closely.
• Split each answer into separate claims.
• Check those claims against a reliable source.

Four steps for each claim, times several claims, inside one minute. The rater judges what is visible in the time available. Longer, confident, well-structured answers feel better. A long wrong answer beats a short correct one when nobody checks the facts.

Some raters paste the pair into GPT-4 and copy the verdict. The judge and the model stop being independent, and self-preference rides back into the training data.

The work is often outsourced, and the pay runs low against the cognitive and emotional load. Culture and religion shape what a person calls a good answer. RLHF is the last high-impact step in the pipeline. When most raters share a region or a faith, the outputs drift toward those values.

The open question is whose preferences count. Crowd-worker demographics set part of the answer. Company policy and the bias of an AI judge set the rest.

AI judges agree with humans about as often as humans agree with each other

A strong model reads two answers and picks one, faster and cheaper than a person. Experiments put its agreement with human labels at about the level two humans reach.

Large feedback sets follow from it. UltraFeedback is one, and systems such as Zephyr and Tulu train on judgments from an LLM.

The preferences arrive in two kinds, and pipelines mix them. Off-policy preferences cover outputs from many models. On-policy preferences cover the outputs of the model being trained. The mix decides which part of the answer space the reward model has ever seen.

Both sides of that pipeline prefer longer answers. Humans favor long, list-like answers, and LLM judges favor them too. The preference holds even when the long answer carries more hallucinations. RLHF then pushes the model toward verbosity.

The evaluation note found length bias deciding leaderboards without ever appearing on them. It arrives here through the training door. Track length and style in the training data and in the evaluation. Treat any win that moves with length as unproven.

One score never settles a post-training question. Many benchmarks barely move with answer length. Chat evaluations move with verbosity and vibes, and MMLU-style scores miss style and safety. A working pipeline reads more than one measurement before it ships.

DPO gets most of PPO's gain with a supervised objective

The true reward is never observed. What arrives is a set of pairwise choices, each one saying that this answer beat that one.

Bradley-Terry turns those choices into a score. Each answer gets a number. The chance a person picks one rises with the score difference, through a logistic function.

Fit that model by maximum likelihood on the preference data and you have a reward. Then push the policy toward a higher score. The policy gradient raises the probability of high-reward answers and lowers the probability of low-reward ones.

That loop is unstable on its own. Variance is high, and a policy that trains again on old data drifts. PPO holds it together. An advantage subtracts a baseline, clipped ratios bound each update, and a KL penalty keeps the policy near the reference. It works, and it is a lot of machinery to build and tune on a large model.

DPO removes the loop. Under KL regularization the optimal policy and the reward have a closed-form relation. The reward becomes a log-ratio of the policy against the reference.

Training then runs on chosen and rejected pairs directly. Raise the log probability of the chosen answer and lower the log probability of the rejected one. The explicit reward model and the PPO loop both drop out. The objective looks supervised, and open research uses it heavily because it works.

The preference set is the objective now. Whatever the labels rewarded, the gradient rewards.

The Builder Test

Name the proxy before you trust an improvement. Take the last preference set or instruction set you shipped.

Write two lines about it. The first names what a label rewarded, whether that is length, structure, politeness, or a citation at the end. The second names what no label checked.

Then write the sentence a model can follow to win the first line while failing the second. If you can write that sentence in a minute, a training run finds it faster. Fix the data, or add a check that a program can run. Measure the fix against the behavior you named.

What Carries

Post-training reshapes behavior with a small amount of data. The same step that makes a model useful and polite installs hallucination and bias without a signal. Every choice of data, labeler, and feedback rule shows up in how the model speaks, refuses, and invents.

The model learns what the system rewards. Whoever wrote the rubric meant something else. The reward is what you optimize now, so the next question is what happens when you push on it hard.