Loading
Every rollout row in the sentence sits between 0.935 and 0.971 normalized entropy. That is a uniform distribution wearing a ranking, and it shipped as the default view.
Open the instrumentIf a model has sixteen attention heads, showing one is a choice you have to defend. So there is a natural instinct to combine them, and there is a published method for doing it properly.
Attention rollout propagates attention through the network. Average the heads in each layer, add the identity matrix to account for residual connections, renormalise, then multiply the layers together. The result is a single matrix describing how information flows from input tokens to output positions across the whole model.
It is principled, widely cited, and it was this project's default view.
It was also, on this model, statistically indistinguishable from showing nothing.
The decisive idea: a ranking computed from a nearly uniform distribution is still a ranking. It will order the words for you, confidently, and the order will be close to arbitrary.
Normalized entropy is the tool. Take a row, treat it as a probability distribution, compute its Shannon entropy, divide by ln(n). The result is 1.0 for a perfectly uniform row and approaches 0 as mass concentrates.
For the sentence "Enter an English sentence to inspect how attention differs across heads", every rollout row measured between 0.935 and 0.971.
Normalized entropy of 1.0 is a uniform row — one that says nothing. Rollout mixes every layer and every head, and on a four-layer model the result sits near the top of that scale for every word in the sentence.
Switch between rollout and the selected head for the same word. The rollout row is nearly level — the bars barely differ. The head's row has structure you can see without measuring anything.
The figure above is a different sentence, and it behaves the same way. Across all nine of its words, rollout entropy stays between 0.952 and 0.973, and the peak of a rollout row never gets past 1.76× uniform. The heads the instrument selects for those same words run 3.52× to 6.49× uniform, at entropies of 0.443 to 0.799.
For are, side by side:
| View | Peak | × uniform | Entropy |
|---|---|---|---|
| Rollout | 0.141 | 1.27× | 0.963 |
| Head L1H4 | 0.499 | 4.49× | 0.734 |
A peak at 1.27× uniform is not a finding. It means the winner beat an even split by a quarter, on a row of nine words, and small changes anywhere would reshuffle the top three.
This is not a bug in rollout, and not a defect in the forward pass. It is a documented property of the method interacting with a shallow model.
Rollout adds the identity matrix at every layer to model the residual stream, then multiplies. Each multiplication mixes every position with every other position a little more. After enough layers, the product approaches something close to uniform — the mathematical signature of a random walk losing its starting point.
On a twelve-layer model, rollout has enough depth for structure to survive the mixing. On four layers, the smearing outruns the signal. The method is not wrong; it is being asked to summarise a process too short to have much to summarise.
The defect was not using rollout. It was showing rollout by default and letting a visitor read a ranking out of it.
The interface presented an ordered list with three-decimal weights. Everything about that presentation said "these words are ranked by importance." A reader has no way to know the ordering is barely above noise. The numbers are true — that rollout row really does have those values — and the impression they create is false.
Same failure mode as the previous post, in a new place: correct arithmetic, misleading presentation.
The fix was to measure how peaked a row actually is and let that drive what gets shown. Rows are now classified by their ratio: below 1.6 is flat, 1.6 to 3 is diffuse, above 3 is selective. Selection prefers a head that gives a selective row for the focused word, and the shape verdict is displayed alongside, so a reader can see when the tool is telling them the row does not say much.
Rollout is still available, still labelled, still exactly correct. It just no longer opens by default pretending to be a ranking.
It stays the honest choice for the recorded result that opens the scene, because that export carries head-averaged rollout only — the individual heads were not saved. Showing a per-head view there would mean inventing heads that are not in the file.
And it remains the correct answer to a question nobody was actually asking: across the entire network, including residual paths, how does information flow from this input to that output? That is a real question with a real method. It is not the question a visitor clicking a word is asking, which is: what does this word connect to?
Matching the statistic to the question is most of interpretability. The rest is admitting when the statistic you have does not answer the question you were asked.
Next: how do you test a model when you cannot look at the answer and tell whether it is right?
More to read
A median 24% of every displayed row had already been deleted before you saw it. In the worst case, 95%. The arithmetic was right the whole time.
I wrote the code and the tests from the same wrong idea, and they agreed with each other perfectly. Everything was green.
The word-embedding table is 31,254,528 bytes. A twelve-word sentence needs about 20 KB of it. HTTP Range makes that the only part that travels.
