Data Is the Model Diet

Two 1B models trained on the same 100B tokens land several points apart. One draws from raw Common Crawl. The other draws from a filtered pool like DCLM. The filtered model scores higher across knowledge and math benchmarks, on the same compute.

Two models on the same token budget land points apart, which is why labs stay vague

That gap is data quality alone, and it grows with the size of the run. The corpus sets the ceiling. Extraction, filtering, deduplication, and mixture decide where the ceiling sits, and the architecture works underneath it.

Data decisions also cost almost nothing next to the training run. They are the cheapest large effect available to a team.

Model papers describe the architecture in detail and stay vague about the corpus. LLaMA 3 names broad sources and rough filtering, and stops there. Two reasons keep the recipe private.

• The mix and the filters drive quality, so the recipe is the advantage.
• Fewer disclosed specifics leave fewer targets for a lawsuit.

The overview said every stage prices the next one. Data is the stage that prices all of them. Foundation models cut manual labeling and moved that work into selection, cleaning, filtering, and mixing.

Data work also spreads across a large team. Architecture work stays with a small core. When compute and architectures converge, the corpus is what one lab has and another does not.

Common Crawl is a biased snapshot, and the extractor is a quality lever

Common Crawl is a non-profit that started crawling in 2007. It runs a crawl every month, and each one covers billions of pages. Coverage is incomplete and skewed, and the result is a sample of the web.

The crawler starts from a seed set of URLs and keeps a frontier queue. It fetches in parallel, follows links, and adds what it finds. Heuristics decide revisit frequency. Robots.txt holds down the load.

Each crawl ships in two forms. WARC keeps the raw HTTP responses. WET keeps extracted text, and that extraction is lossy. The same crawl also carries harmful content, heavy duplication, and gaps where important sources are missing.

Turning HTML into text is a major quality lever. Extractors disagree about what is content and what is boilerplate. Swapping one extractor for another moves benchmark scores by multiple points.

Teams drop the default WET text for that reason. RefinedWeb ran Trafilatura over Common Crawl and argued that filtered web alone is enough. NeMoTron CC later switched to jusText to keep more usable text.

The evaluation note named this failure. A pipeline change moves a benchmark by multiple points, and no leaderboard can say which stage bought the move. Report the extractor next to the score. Without it the number has no owner.

Every filter encodes a belief about what good looks like

GPT-2 built WebText out of a proxy for worth reading. Take Reddit posts above 3 karma, follow the outbound links, and download those pages. That gave about 8 million pages and about 40 GB of text. The belief inside the rule is that upvotes track quality.

OpenAI kept WebText private. OpenWebText is the open reproduction, and many teams train on that instead.

CCNet deduplicates, runs language identification, then scores each document against Wikipedia statistics and keeps what resembles Wikipedia. Wikipedia carries sourced facts, few recipes, and little first-person advice, and its notability rules keep out niche topics. A filter tuned to it inherits that shape.

C4 trusts grammar. Google built the Colossal Clean Crawled Corpus from one Common Crawl snapshot for T5. It keeps English lines that end in sentence punctuation and drops pages under three sentences. The rules also drop pages with bad words and lines with braces, which removes most code with the spam.

The two blind spots run opposite. CCNet drops useful text that reads differently from Wikipedia. C4 keeps spam that is well formed.

GPT-3 replaced hand rules with a learned filter. Its mix was processed Common Crawl, WebText 2, two books corpora, and Wikipedia. Positives came from WebText, Wikipedia, and books, and negatives came from random Common Crawl pages. The classifier kept the crawl pages that looked like the positives. The whole recipe came to about 400B tokens.

That pattern is now standard. Define what good looks like, train a classifier on it, then mine a better subset from a huge pool. LLaMA trained its classifier to match sources cited by Wikipedia and reached about 1.2T tokens.

Gopher went the other way. DeepMind wrote quality rules by hand and used SafeSearch to cut harmful content. It avoided model-based filters for one reason: a weak filter model stamps its own bias onto the corpus. MassiveText held MassiveWeb, C4, books, news, GitHub, and Wikipedia, about 10 TB in all. Gopher trained on about 300B tokens.

Open reproductions made the filters inspectable. RedPajama v1 rebuilt the LLaMA mix with data and code released, and Cerebras published a deduplicated subset near 627B tokens. RedPajama v2 processed about 30T tokens and attached several quality signals so teams can run filter experiments.

RefinedWeb pairs strong extraction with Gopher-style rules and fuzzy deduplication. It holds about 5T tokens and released about 600B of them publicly. FineWeb runs rule-based filtering, basic anonymization, and deduplication across every Common Crawl dump and stops near 15T tokens. It exists as a base layer for teams that filter harder.

DataComp made filtering itself the measured object. The DCLM pool processes every dump into about 240T tokens. DCLM Baseline cuts that to about 3.8T with rule-based and model-based filters.

The model-based filter is a FastText classifier. Positives are OpenHermes and ELI5, negatives are random FineWeb samples, and the classifier scores the whole pool. DCLM Baseline beat RefinedWeb and other datasets on many benchmarks, and OLMo adopted it. Filtering now runs with a model in the loop. The filter carries the biases of the model that built it.

3.8T tokens is too small for a very large run. NeMoTron CC kept the DCLM idea and went back for tokens that DCLM threw away. A large NeMoTron model rates educational value, and the team distills that rater into a faster classifier. The DCLM classifier supplies a second score.

NeMoTron CC then buckets documents by score and samples across all buckets to hold diversity. Some pipelines also rewrite low-quality documents into cleaner text with an LLM. From the high-quality ones they generate questions, summaries, and key-information pairs.

The result is about 6.3T tokens from Common Crawl, beating DCLM and FineWeb on average benchmarks. A 1T-token high-quality subset performs best of all.

Karma, Wikipedia similarity, sentence punctuation, and an educational-value rating are four claims about good writing. A repeated document teaches memorization. A bad extraction teaches noise. The model learns the claim you encoded.

An open dump is an attack surface

Wikipedia is heavily edited and multilingual, and it publishes downloadable dumps on a schedule. An attacker can time a schedule.

An attacker edits a page before the dump runs. The edit lands in the snapshot. A revert on the live site cleans the page. The dump still carries the poison.

A model trained on that dump learns the association the attacker planted. Nobody auditing the live page can see the text that taught it.

Special sources buy capability and carry licenses

Stack Exchange ships a question, an accepted answer, and votes. A question comes in and an answer comes out, which is the shape of instruction following. The votes are a quality signal that costs nothing to compute. Dumps exist, and commercial use can need a license.

GitHub is the code source, and most of it needs sorting. Many repositories are tiny, abandoned, forked, or duplicated. Three decisions build the corpus: what counts as code, which licenses pass, and how forks get deduplicated.

The Stack read repository names from GitHub Archive and cloned about 137M repositories. It kept the permissively licensed ones, deduplicated them, and produced about 3.1 TB of code. Every source runs that path: live service, snapshot, processed training set.

Papers add dense scientific text. PubMed Central publishes open-access biomedical work, and Semantic Scholar aggregates across many sources.

Books buy long context. Project Gutenberg holds tens of thousands of public-domain books, and PG-19 comes out of them. Books3 came from a shadow library and carried copyrighted works. It was removed after complaints.

LibGen and Sci-Hub host large collections outside copyright, and lawsuits claim that some models trained on them. That is the second reason for the vagueness, and it is the one with a court date.

The Pile mixed OpenWebText, Stack Exchange, ArXiv, PubMed Central, GitHub, Gutenberg, and Enron emails. It came out larger than the GPT-3 training data. A curated source is easier to control and reuse than a crawl, and it leaves a fingerprint. The Enron mail shows up in how those models write email.

Dolma stacked processed Common Crawl, The Stack, C4, Semantic Scholar papers, Gutenberg, and Wikipedia into about 3T tokens. It ran English language identification, quality filters that started without classifiers, a classifier for harmful content, and deduplication.

Dolma also pulled Reddit submissions and comments through a project that later lost its public API. A source can close after the corpus depends on it.

Copyright and platform terms set the real boundary

Copyright attaches the moment a person posts text. It needs no notice and no registration. It covers expression and leaves ideas free, and the terms run for decades. Most of the web is protected by default.

Registration matters when someone sues. Using protected text needs a license or a fair-use argument that holds. Platform terms bind on their own. A work under an open license can still sit behind a rule against bulk downloading.

Creative Commons keeps the copyright and grants reuse under conditions, which is how Wikipedia is licensed. Direct deals between builders and content owners cover more. Most teams end up mixing licensed data, open data, and a fair-use claim.

Courts weigh four factors.

• Purpose and character, which favors a use that changes the work, and research over commerce.
• Nature of the work, which protects creative writing more than factual writing.
• Amount used, and training copies whole works.
• Effect on the market, which asks whether the new use replaces the original.

A court found Google Books scanning and snippet display fair. The scans served a new purpose, and the snippets did not replace book sales. Builders argue training is the same kind of use, since a model learns patterns and not the text.

Training is the harder case. It copies whole works, and models can memorize and reproduce what they copied. Courts are still deciding.

Price the legal exposure with the compute, before the run starts. A corpus you cannot defend is a run you cannot ship.

Mid- and post-training buy targeted skills for a small share of the tokens

Pre-training takes the huge noisy web corpus. Mid-training takes smaller, cleaner slices aimed at math, code, reasoning, or long context. Post-training does instruction and chat tuning, often with reinforcement learning for helpfulness and safety.

A base model comes out of pre- and mid-training. Post-training produces the chat model that obeys instructions and holds a conversation. The boundaries blur, some pre-training data already reads like instructions, and some pipelines run more than three stages.

Mid-training is a small fraction of the total tokens. AI2 built theirs from filtered slices and instruction sets in the FLAN style. Synthetic data covers math, reasoning, and long context.

The efficiency note priced attention against sequence length, and that price is why long context arrives late. Teams train mostly on short sequences and add long documents at the end. Books, long articles, and synthetic long sequences fill that stage.

Older NLP kept a separate format for every task, and instruction tuning collapsed them into one. SuperNaturalInstructions reformatted more than 1,600 tasks into instruction prompts and outputs. FLAN did the same across many NLP tasks. Both lift benchmark numbers. The prompts stay templated, and user language is messier.

Self-Instruct and Alpaca start from a seed set and let a strong model write both the instructions and the answers. Vicuna trained on ShareGPT logs, which showed that real conversation lifts chat quality. Self-chat and evol-instruct raise the difficulty step by step. Another pattern pulls question-and-answer pairs out of web text with an LLM.

OpenHermes is a curated mix of instruction and chat sets. It also serves as the positive class for quality classifiers, which runs the loop back to filtering.

LLaMA 2 Chat leaned on human-annotated instruction and chat data, and it beat large synthetic sets. Human data is slow and expensive, so teams stretch it with RLHF. Teams also have to keep annotators from using GPT-4 in secret.

LLaMA NeMo post-training mixes public chat sets like WildChat with synthetic conversations from open models. It adds reasoning traces from models like R1 to teach chain-of-thought output. The rest of a post-training set is licensed chat transcripts and data shaped for safety.

The Builder Test

Sample a hundred documents your filter kept and a hundred it dropped. Read them before you trust the mixture.

If the kept pile surprises you, the corpus teaches that surprise to the model. If the dropped pile holds text you wanted, the filter carries a belief you did not choose.

What Carries

The model cannot become better than the signal you give it. Every filter is a claim about what that signal is.

Size alone is a bad measure of a corpus. More tokens make a worse model when they are duplicated, poisoned, low quality, legally risky, or weighted badly. The field still runs on heuristics, and a better pipeline still buys a large gain.

Filtering decides what stays. Deduplication decides how often what stays repeats, and it is a separate job. The mechanics of doing both cheaply are the next note.