Data Is the Model Before the Model

The dataset is the first behavior specification a model receives. WebText picked its documents by following outbound Reddit links and treating that as a proxy for quality. That choice landed before a single training step ran.

Data work sits upstream of almost every behavior people later attribute to the model. Overrepresent one domain and the model learns that style. Leave duplicated text in and the model can memorize it. A filter that removes some communities harder than others hands the model that skew.

The Model Arrived Through A Supply Chain

Someone crawled the web, someone wrote the filter, and someone set the mixture weights. Someone else chose what to write down about all three, or chose to write down nothing. Those four decisions reach the user before any prompt does.

WebText, OpenWebText, C4, the GPT-3 mixture, and The Pile each encode a different set of collection and filtering choices. Underneath those sits one more choice: whose text becomes training material. The people who wrote that text are not in the room for it.

Every corpus carries duplication, quality variation, demographic gaps, and domain bias into the training run. None of that arrives in the finished weights with a label attached. I do not trust a behavior explanation that stops at the architecture.

Every Corpus Encodes A Proxy For Quality

Nobody reads a web crawl end to end. So each corpus builder picks a proxy and inherits whatever it correlates with.

WebText used outbound Reddit links as its proxy. That proxy measures the taste of the people posting the links, and the corpus takes on that taste. OpenWebText rebuilt a similar corpus in public, so the proxy outlived the team that chose it.

C4 filtered Common Crawl, and a filter is a value-laden transformation. What the filter drops leaves no mark on the model and no mark on the benchmark. Whole languages can leave a corpus that way.

GPT-3 mixed Common Crawl with WebText, books, and Wikipedia at chosen weights. Chosen is the load-bearing word. A source can sit in that mixture louder or quieter than its raw size, and a person picks the number.

The Pile emphasized breadth across domains: code, academic text, web text, and books. Every domain in the mixture is a register the model gets cheaply. Code is the clearest case, because code in pre-training makes code cheap to imitate.

Source, Filter, Mixture, Deduplication

More data is not automatically better. Repeated, low-quality, or misweighted text makes the model worse and the system harder to defend. Four levers decide which of those you get.

• Source decides domain coverage.
• Filtering decides what disappears.
• Mixture weights decide what becomes normal.
• Deduplication decides which text stops shouting.

Each lever costs something in a different part of the system. Duplicates left in the corpus become the memorization the security note describes. Filtering decides which communities thin out, and that thinning is the disparity the first harms note measured. Mixture weights change data quality, and quality bends the curve the scaling-laws note fits.

Pre-training Absorbs The Incentives Before Any Later Tuning

A dataset carries more than text. It tells the model what is common, what is authoritative, what is safe, and what is cheap to imitate. Pre-training absorbs those incentives before any instruction tuning or preference tuning gets a vote.

Web pages, books, code, academic papers, forums, and synthetic examples do different jobs in the finished model. The design question is which behaviors must become easy. The mixture weights are the answer you gave, whether or not you wrote the question down.

Before You Debug Behavior, Inspect The Mixture

The overview note asks you to name the layer that did the work. When the surprise is about what the model knows, imitates, or repeats, the layer is the corpus.

A user asks why the model knew that, imitated that, or leaked that. If you cannot trace the likely data path, you have a governance gap. Prompt changes do not close it.

Datasheets and dataset cards are the minimum operating manual for a corpus that shapes model behavior. A datasheet records sources, collection process, filtering, intended use, limitations, and known risks. Those six fields answer the traceability question one field at a time. Read them back and you can name the likely data path, or see which field is empty.

Data governance for an LLM system means provenance, deduplication, filtering rationale, contamination checks, privacy review, and a refresh strategy. Each one is a document somebody keeps current. Without them, debugging behavior turns into guessing about a corpus nobody can see.

The Builder Test

Pick one behavior your system shows this week. Trace it to a source, a filter, a mixture weight, or a duplicate. Write that trace in one line and keep it with the model.

If you cannot finish the line, you have found the missing document. Go write it before you touch the model.

What Carries

"We trained it on a lot of text" is where most explanations stop. Quantity without provenance is an admission that nobody can name where the behavior came from.

Write the mixture down while you build it, and keep that document next to the weights. A corpus that can be memorized is a corpus that can be extracted.