Security Begins With Memorization

Extraction attacks turn generation into a search for memorized text. Repeated sampling, targeted prompts, and a search over completions can recover rare sequences the model saw in training. Once a secret is trained in, you cannot cleanly delete it.

That moves the security question off the database and onto the model. Some of the text a model reads is sensitive, copyrighted, private, or recoverable by an attacker who knows how to ask. The weights join the list of places where your secrets live.

A Model Is A Compressed Artifact Of Its Training Data

Memorization risk rises with the copy count. A rare string that appears many times is distinctive enough to stand out and repeated enough to learn. That is the worst pair of properties a secret can have.

The data note in this unit puts the corpus upstream of the behavior, and duplication is one of the ways that shows up here. A crawl collects the same passage from many pages, so the copy count is set before anyone picks a training mix. Deduplication is the control, and it works only while the data is still a file you can edit.

Treat the model as a data-bearing component. Secrets that pass through training data, logs, prompts, retrieval corpora, or fine-tuning sets do not stay where you put them.

Extraction Turns Generation Into Search

An extraction attack needs no new algorithm. The attacker samples the model many times, steers it with targeted prompts, and searches the completions for text that looks memorized. The model does the retrieving.

Membership inference asks a narrower question. It tests whether one specific record was likely in the training data, and it never has to reproduce a line of that record. Membership in a corpus can be the sensitive fact on its own.

Both attacks share one frame. Name what information entered training and how likely extraction is. Then say which prompts or access patterns make extraction easier, and which controls cut the damage. Privacy work starts when those four answers come from your own system.

The Deployment Wrapper Decides Whether A Flaw Becomes A Breach

Memorization is one entrance. Prompt injection, tool misuse, jailbreaks, a compromised model supply chain, poisoned training data, and leakage through logs use the same stack. The wrapper you build around the weights decides how far each of them travels.

Authentication decides who can query the model. Rate limits raise the cost of the repeated sampling that extraction depends on, and audit logs make the attack visible while it runs. Retrieval permissions bound the private context the model can reach at answer time. Incident response sets how fast you cut access after someone reports a leak.

Red-team prompts are how you find out which of those controls holds before an outsider does. Each control is also a threshold, and the first harms note named what a threshold does. It picks which error you eat. A tight rate limit blocks a real user. A loose one lets the sampling run long enough to find something.

Split Instructions From Untrusted Content

Prompt injection turns serious the moment the model reads untrusted text while holding tools or private context. A fetched web page becomes a privileged instruction, because the model has no structural way to tell your intent from the text it fetched.

Split the two channels in a place the system can enforce. Instructions come from you and from your own code. Fetched pages, uploaded files, and tool output stay data the model reads, even when the text inside them sounds like an order.

Then cap the side effects. A demo shows a model that can do anything it is asked. A product gives the model the short list of actions you have priced, so a successful injection reaches less than the demo suggests.

Some Secrets Must Never Enter Training

Controls split by when you can still apply them. Data minimization, deduplication, and differential privacy for sensitive data are training-time choices, unavailable the day after the run finishes. Monitoring, canaries, and strict tool permissions are what you have left.

After the run, what the model holds is fixed. You can filter the output, refuse the prompt, and revoke the key. The string is still in the weights, and no delete you run reaches it. Privacy work that waits for a leak starts after the cheap moment has passed.

The Builder Test

Plant the canary before you need it. Put a unique string into the training data or the retrieval corpus. Pick one that nobody types by accident, and record where you put it. Then attack your own system.

• Sample the model repeatedly with targeted prompts, then search the completions for the canary.
• Repeat the attack at every access level you ship, from anonymous visitor to internal tool.
• Run a membership inference check against one record you know sits in the corpus.
• Write down which log or filter caught the attack, and how long the catch took.

The access level is part of the result. The same prompt at anonymous and at admin returns two different answers, and both belong in the record. I prefer to find my own canary in a completion, before a stranger finds it for me.

What Carries

Training data is an input surface. Generated text is an output surface. Both of them leak, and a control that watches one of them misses the other.

A secret you cannot delete becomes an obligation you cannot satisfy. The legality note takes up the deletion duty and the removal path. Before the next run, ask whoever owns the corpus which strings a removal request must reach, and whether you can reach them.