Continuity checking, as most tools do it, is a fact-matching problem. Mara's eyes were grey in chapter two; they had better be grey in chapter forty. That part is tractable, and Guardian runs fourteen zero-LLM rules plus four model calls over it.

But a novel is not a database of true things. Half of what drives a story is a character being wrong — believing the letters burned when the letters are in a drawer, intending something in chapter nine and abandoning it by chapter ten. A checker that only knows the world state will "correct" the false belief and report a contradiction that is actually the plot.

So we needed a second ledger: not what is true, but what each mind holds.

This post is about building that ledger, measuring it, discovering the measurement was wrong, and what the honest numbers turned out to say. It is the engineering behind why we point AI at verification rather than generation.

Four dimensions, one extra call

We took the shape from EvolvTrip, which models character psychology across belief, emotion, intention and desire. Ours are belief_about, feels_towards, intends_to and desires_for. The first two take a target (A believes something about B); the last two do not.

It is a separate LLM call from the world-fact extractor, which was not the original plan. The re-spec said it should ride the existing call, and there were three reasons not to:

The world-fact prompt excludes mental states by name — "anything a character merely claims, dreams, fears, or imagines". That exclusion is a precision guard that currently works. Relaxing it is not a marginal edit.
Recall degrades as one prompt's schema widens. The world call had already spent its one extra field.
Observability — who could have known what — is a whole instruction block that is irrelevant to world facts and expensive to state twice.

The cost is one more call per chapter. That is affordable for a specific reason, which is the rest of this post.

It ships dark

Every row the extractor produces is written to the database and read by nothing. Not the chat context, not Guardian, not the character cards.

This is deliberate. Unreviewed LLM extraction of what a character believes is a dangerous thing to let steer generation: get it wrong and the assistant confidently builds on a belief the character never held. So the rows accumulate against a future in which we can show they are right, and the gate that releases them is per category, not one switch.

Opening a category is deleting one line from a list. The rows it already wrote become visible — they were being collected against exactly that moment.

The benchmark is authored, not annotated

To measure this we wrote 40 fixtures, 20 in English and 20 in Chinese, across ten trap families. They are not annotated passages — each one is written to make a specific mistake attractive, so the gold is known by construction and reviewing a fixture means checking that the trap is fair, not re-deriving the answer.

A few of the families:

false-belief — the passage states the world one way and the character's belief the other. An extractor that "corrects" the belief has failed.
off-page-learning — the narration tells the reader something the character was not present for. Marking it observable is the mistake.
overheard-vs-absent — the mirror. The character is positioned to know, by perception or by testimony. A model that marks everything unobservable passes the family above and fails this one. Neither family is scoreable alone.
narrator-only and abandoned-intent — these expect zero facts. Recall is the weak axis, and the cheap way to score well on recall is to extract everything; these make that strategy cost precision.
nested-belief — second-order belief, the case OpenToM and FANToM are built on, and the one that separates theory of mind from sentiment analysis.

Then we found the benchmark was lying

We rebuilt the whole suite in August 2026 and voided every precision number we had published to ourselves before that date. Three separate faults, each of which made the score better than the system:

The scorer matched on keys, not content. It paired model output to gold on (category, subject, target) and explicitly not on what was asserted. So a model that reported "Mira believes the letters are safe" — the exact failure false-belief-01 exists to detect — keyed identically to gold and scored as a clean true positive. Half the gold in the suite is belief_about.

observable was defined one way and graded the opposite. Six of eight fixtures carrying it were scored against the inverse of the definition in the prompt.

Seven of sixteen English fixtures were filed under the wrong family — an arity test sitting under false-belief, four behaviour-shown fixtures under overheard-vs-absent with nothing overheard. The per-family table and the README that described it were both wrong.

The fix for the first one is the interesting one. The bench now matches on the key to find candidate pairs, then asks a judge whether the model's statement asserts the same mental content as the gold. Disagreement on content is its own bucket, and it costs precision and recall — the gold row was not produced, and a false claim about a mind was.

What honest measurement said

The number we had been quoting was a pooled precision of roughly 28–30% across all four categories. Against the rebuilt suite, per category:

feels_towards — precision 85.7%, recall 75.0%. Released.
belief_about — 40.9% / 50.0%. Dark.
desires_for — 22.2% / 50.0%. Dark.
intends_to — 18.5% / 62.5%. Dark.

The pooled figure hid the 85.7%. Three weak categories were holding the one that worked shut, and no amount of arguing about per-category gates in the abstract would have shown that — the measurement did.

feels_towards was released on 2026-09-17 against a floor of 70% precision with at least three true positives. The other three are still off, and will stay off until they clear the same bar. Emotion being the strongest is also the result EvolvTrip reports, which is mildly reassuring and not evidence of anything on its own.

A drop is a measurement, not a void

One habit this built that outlived the feature. The extractor refuses model rows for about ten reasons — confidence below floor, subject not in the roster, evidence not verbatim, wrong arity for the category, target identical to subject. Until August those refusals were a bare continue. No counter.

Which meant the bench printed MISSED — indistinguishable from a theory-of-mind failure — for rows the model had returned correctly and our own plumbing had eaten.

Every refusal now lands in a histogram with its reason and the raw row. Two of the reasons are scored as false positives rather than disappearing. And the two early returns that skip the model entirely set an explicit skipped marker, because "never called the model" must not read as "found nothing".

The fixtures, the scorer and the trap families are open source in narrative-bench. If you are measuring something similar and your numbers look good, the three faults above are worth checking for first. Ours looked good too.