Mind Map: The Sentence You Type Is Not What the Model Reads

For weeks, Mind Map was drawing attention arcs for a sentence nobody had written.

If you typed It costs £5, the model did not receive £5. It received £ and then 5, as two separate things with no more relationship than any other pair of neighbours. The vocabulary has an entry for £5 — row 27,813, sitting there since 2018 — and the tokenizer walked straight past it.

Nothing looked wrong. The interface showed words, the arcs had plausible weights, the numbers summed correctly. The arithmetic downstream was flawless. It was flawless arithmetic performed on the wrong input, which is the most expensive kind of correct.

The decisive idea: A model does not read your sentence. It reads a list of integers, and everything it can possibly notice was decided before the first layer ran.

The Gap Between What You Type and What Arrives

Between your keyboard and the first matrix multiply there is a translation step, and it is lossier and stranger than almost anyone expects.

BERT does not have a word for every word. It has exactly 30,522 entries, fixed when the model was trained and unchangeable afterwards. Every sentence in every language you could type has to be expressed in those 30,522 symbols, the way a printing press with a fixed drawer of type has to set any book from the letters it owns.

When a word is in the drawer, it is used whole. When it is not, it gets rebuilt from fragments.

The rebuilding rule is called WordPiece, and it is aggressively simple: take the longest piece from the front that exists in the vocabulary, cut it off, and repeat on what remains. Every piece after the first is written with a ## prefix, which marks it as a continuation — a fragment that can only ever attach to what came before.

That greedy rule is why the fragments so rarely match the syllables you would have chosen:

hen + ##s is a decomposition a person would endorse. uno + ##pen + ##ed is not — there is no sense in which "unopened" is built from uno and pen. The tokenizer is not trying to find meaning. It is trying to cover the string with the fewest longest pieces it has, and it will happily hand the model uno and pen because those cover the characters.

This has a consequence worth sitting with: the model never sees the word "unopened." It sees three fragments and has to reassemble any notion of that word across three positions using attention. That reassembly is real, and it works better than it has any right to — but it is work the model is doing, not a fact it was given.

The Bug

Here is the rule I got wrong.

HuggingFace's tokenizer, before WordPiece ever runs, does a first pass that splits text on whitespace and on punctuation. My implementation split on whitespace and on anything that was not a letter or a digit.

Those sound like the same rule. They are not, and the difference is exactly the set of characters that are symbols rather than punctuation: £, °, ¹⁄₂, emoji. HuggingFace does not split on those. Mine did.

The divergence was invisible for a specific and uncomfortable reason. ASCII symbols — $, %, +, =, &, * — fall inside HuggingFace's punctuation ranges, so both implementations split them identically. Every test anyone thinks to write uses ASCII. The bug lived entirely in the non-ASCII tail.

Nine vocabulary entries could not survive being typed:

°c (6362)   °f   £1   £2   £3   £5   £10   £100   ¹⁄₂

And two emoji, which HuggingFace turns into a single unknown token, became two separate unknowns.

Nine entries out of 30,522 is 0.03% of the vocabulary. It is also 100% of the sentences containing them, and every one of those sentences was getting an attention diagram computed on an input the model would never have been given.

Why Testing Did Not Catch It

The tests passed because the tests and the code shared an assumption.

I had written the tokenizer, then written tests for the tokenizer, from the same mental model of how tokenization works. When your model of the rule is wrong, you write a wrong implementation and a wrong test that agree with each other perfectly, and everything is green.

What broke the deadlock was building a second tokenizer that shared nothing with the first — different language, different author-day, written from the specification rather than from the code, running against the original weights in fp64 NumPy. When two independent implementations disagree, one of them is wrong, and you no longer get to be comfortable.

It disagreed on £5. The second one was right.

I want to be careful here, because that oracle has an ego problem it has earned the hard way: it was also wrong, once. Its first version lacked the step where HuggingFace pads every CJK character with spaces so each becomes its own word. It disagreed with the shipped runtime on 中文, and that time the runtime was correct. A committed fixture settled it.

An oracle nobody validated is just a second opinion. That one is now checked against a reference PyTorch run before it is allowed to judge anything, and the check runs before every comparison.

The strongest guard turned out to be the least clever one. Rather than list the words I could think of, take every one of the 30,522 entries in the vocabulary, feed each back through the tokenizer on its own, and require it to come back as itself. A vocabulary entry that does not survive a round trip is a word the model owns and can never be handed. That test found the class of bug rather than the instance, and it is the reason I know there is no tenth entry hiding behind the nine.

What This Means When You Read an Attention Diagram

Three things follow, and they change how you should look at every visualisation in this series.

The units on the screen are not words. When Mind Map shows you hens, it is showing a word assembled from two token positions, and the weight next to it is those two positions merged. That merge is a choice — an average across the query's pieces, a sum across the target's — and a defensible one, but it is a layer of interpretation sitting between you and the model.

Some inputs are genuinely outside the model. Type an emoji and you get [UNK] — not an approximation, but the absence of one. The model receives a token meaning there was something here and I have no idea what. Any arc drawn to it is real attention paid to a shrug.

Length is not what you think. The interface caps input at 24 tokens, not 24 words. Water boils at 100°c. is five words and ten tokens. The limit is not knowable until the vocabulary has downloaded, which is why the tool reports it honestly after the fact rather than predicting it beforehand.

Tokenization is usually taught as a preliminary — the boring step before the interesting part. It is the step where the model's entire perceptual world is fixed. Everything the next twelve matrix multiplies can possibly notice about your sentence has to already be present in that list of integers.

Get it wrong and the rest of the system will carry on producing beautiful, confident, well-formed answers to a question you did not ask.

Next in this series: what a single attention head actually computes — three matrices, one softmax, and no metaphors.