Attention Routes Context Instead of Compressing It

Attention replaces one compressed state with a different context vector for every step. The mechanism is retrieval.

A fixed image vector asks one representation to serve every word

A captioning decoder that reads one fixed image vector asks that vector to serve every word it generates. The word dog and the word ball need different regions of the same image. One summary cannot hold both at full strength.

The recurrent networks note priced that compression. One hidden state gets rewritten at every step, and everything the last word needs has to survive the trip. Attention lets the decoder build its context after it knows what this step is asking for.

The change has a price. The decoder now keeps every region available and scores all of them at every step.

Queries ask, keys advertise, values answer

A query states what this step needs. Keys describe what each stored item holds, and values carry the information that comes back.

Scores compare the query to the keys. The score between a query and a key decides how much of that value enters the context. Each step weights the patches it needs and gives the rest little weight.

That makes attention a differentiable retrieval and routing mechanism. The operation is learned and soft, and it usually runs in parallel.

Vision-language tasks buy grounding from it

In image captioning, the decoder looks at the regions that matter as it produces each word. When it generates red car, it must look at the region holding the car, not the sky.

That is grounding, and visual question answering needs it too. The language side gets a mechanism that connects each text decision to visual evidence. The relations it routes over can be spatial, semantic, or cross-modal.

The model is also more flexible than one fixed image vector, and it is easier to inspect. You can read which region fed which word.

Attention weights are evidence about routing and nothing more

Attention weights are useful evidence about routing. A map shows which region the model scored highly for a word. It does not show that the region caused the word.

The visualization note set the rule for probes. A probe earns its place when it produces the next test. An attention map lives under that rule, so read one map, then write down the question it raises.

Attention moved the bottleneck from memory to routing

Self-attention lets tokens attend to other tokens in the same sequence. Transformers stack that operation with feed-forward layers, normalization, residual connections, and positional information.

The old bottleneck was memory compression. The new bottleneck is routing, compute, context length, and data. Those are the limits self-attention inherits when it becomes the backbone.

Architecture is a bet about what structure the data has. Attention bets that the structure worth having is which piece matters at this step.

The Builder Test

Take one generated word from your captioning model. Print the top attention weights and the regions they select.

Then trace the whole path: query, keys, values, weights, and the evidence that came back. Ask whether that evidence supports the word the model produced or a different word nearby. If a nearby word fits the selected region better, you have found the next test to run.

What Carries

Routing decides what the model looks at. Looking is still not understanding.

Routing is the operation that pays here. The next move is to make routing the whole backbone.