Loading
Swap the one adjective that decides what “it” refers to. A reader's answer flips. Not one of the sixteen heads changes its winner.
Open the instrumentThe trophy didn't fit in the suitcase because it was too big.
You know what "it" means. Change one word — big to small — and you know the answer flipped, without effort, because you understand what fits inside what.
This is a Winograd sentence, and it is the cleanest test I know of whether attention is doing what people think it is doing.
The decisive idea: the flip that is instant and obvious to you is invisible to the model. Not weakly present. Not partially detected. Invisible.
The trophy didn't fit in the suitcase because it was too big.
Not one head moved. The flip that is obvious to you is invisible to the model.
Two sentences differing by one adjective. A reader's answer for “it” flips between them. Winners are compared by position, because for two heads the winner is the swapped adjective itself.
Swap the adjective and the model barely moves. Rollout mass on trophy goes from 0.0611 to 0.0616. Mass on suitcase goes from 0.0680 to 0.0740 — it drifts slightly in the direction a reader would go, and both sentences still put more weight on suitcase regardless of which one is correct.
Then the number that settles it: zero of the sixteen heads change their winner.
Not one head, in any layer, picks a different word for it when the answer changes. Whatever these heads are computing, coreference is not among it.
One methodological note, because it nearly produced a wrong claim. My first comparison matched winners by word and reported two heads changing. Both had picked the adjective itself — big in one sentence, small in the other — which is the same position with a different label. Comparing by position gives zero. A metric can be reasonable, defensible, and quietly wrong about the thing you are actually asking.
The model cannot resolve reference. That is a fact about a four-layer encoder, and the honest response is not to hide it but to put it where it bites.
Words whose referent the model cannot recover — it, he, she, they, this, possessives — carry a warning in the ranking when you focus them. Not a disclaimer in a footer; a note attached to the specific row where the limitation applies.
And these sentences are kept out of the suggestion pool entirely. The trophy sentence invites a visitor to ask a question the model cannot answer, and a tool that poses a question it cannot answer looks broken when it is working correctly.
Relative pronouns are deliberately not on that list. Who, which, that link to their antecedent by syntax rather than world knowledge, and the model gets them right: that → Birds 0.860, who → Students 0.700, that → letters 0.680. Warning "attention does not resolve reference" over a row that resolved it correctly would be the same error in the opposite direction — over-caution is also inaccuracy.
Ten posts in, here is the honest inventory.
It tells you: the proportion in which one position's value vector entered another position's output, in one head, at one layer. That is a real fact about a real computation, and this model does encode local syntax you can check by reading — verb to object, preposition to its object, adjective to its noun, relative pronoun to antecedent, subject-verb agreement across a few words.
It does not tell you: that the model understood a relationship. That the weight mattered to the output — Part 9 measured that gap and found it substantial. That reference was resolved. That anything resembling reasoning occurred.
The gap between those two lists is where nearly every misleading attention visualisation lives. Not by fabricating numbers — the numbers are usually real — but by letting a reader carry an impression from the first list into the second.
Five things this project cost me, in the order I learned them.
Correct arithmetic is not correctness. Every defect worth writing about here had the same shape: the math was right and the presentation misled. That failure mode does not show up in tests, because tests check the math.
Your intuition cannot referee numerical output. A subtly broken transformer looks exactly like a working one. Build the second implementation. Validate it before trusting it. Accept that it will occasionally be the one that is wrong.
A check that cannot fail is not a check. Break your own code deliberately and confirm each guard fires. Two stages of my comparison script were printing errors and asserting nothing, and I only found out by perturbing the data to see what happened.
Measure before you build, and set the kill condition first. The norm-weighted idea was compelling, well-cited, and would have taken a day to ship. Measuring took an afternoon and said no. Writing down what "no" looks like beforehand is what stopped it becoming a rationalisation afterwards.
Disclosure is a feature, not an admission. Every place this tool says "here is what this number does not mean" makes it more useful, not less. The version that hid its caveats was less trustworthy and no more impressive.
An instrument that tells you what it cannot see is worth more than one that implies it sees everything. That is the whole thesis, and it took ten posts and one deleted feature to arrive at it.
The instrument is at /projects/mind-map/scene. Type a sentence; it never leaves your device.
More to read
It halves the problem and does not dissolve it. [CLS] carries 0.63× the value norm of a content token — not the near-zero the paper reports. Four layers is not twelve.
I wrote the code and the tests from the same wrong idea, and they agreed with each other perfectly. Everything was green.
Every rollout row in the sentence sits between 0.935 and 0.971 normalized entropy. That is a uniform distribution wearing a ranking, and it shipped as the default view.
