Github: https://github.com/timothyshen/narrative-bench

Writers don't fail because of a lack of ideas.

They fail because they lose control of their own story.

Creader launched nearly three weeks ago with version v0.6.0, presenting a set of core features intended to demonstrate the product's foundational capabilities. Despite this, the application has not yet met the expectations I initially set for it. This discrepancy is not unexpected. Product design often begins with an internal model of user behaviour, yet real-world usage frequently exposes gaps between intention and execution. Rather than treating this as a failure, it serves as a necessary point of reflection.

This article focuses on one of the two central components of Creader: the Guardian system, which can also be understood as a narrative classifier. The conceptual origin of Guardian emerged from repeated observations while reading large volumes of web fiction. A recurring pattern emerges in both reader feedback and author discussions: inconsistencies in plot progression and character behaviour are among the most common sources of dissatisfaction. These inconsistencies are rarely the result of insufficient creativity; rather, they stem from the difficulty of maintaining coherence across an evolving narrative.

The Guardian system was conceived as a response to this gap. Its purpose is not to generate content, nor to replace the author's creative role. Instead, it is designed to provide structural support by performing two primary functions: validating logical consistency against a defined knowledge base, and auditing the narrative for coherence across the text. In this sense, Guardian operates as a form of continuous verification, ensuring that the evolving narrative remains aligned with its own established constraints.

This distinction is important. Whereas many existing AI systems prioritise content generation, Guardian is concerned with preservation: the preservation of logic, structure, and narrative integrity. Writing, particularly in long-form contexts, is not solely an act of creation, but an ongoing process of maintaining coherence within an increasingly complex system.

The initial implementation approached narrative validation through three dimensions: style, consistency, and entity-level analysis. Style analysis focused on extracting a stylistic fingerprint, including tone, point of view, and tense. Consistency validation ensured alignment across character behaviour, plot progression, and stylistic continuity. Entity-level analysis targeted character dialogue, behaviour, and event-level logic. Together, these components formed the initial architecture of Guardian v0.1.

However, during the public launch, the Guardian failed to deliver the expected results. Many detectors were broken or unusable, and the CJK algorithm was incorrect. In one instance, a user writing five chapters encountered repeated duplicate-name warnings while genuine narrative issues remained undetected. This mismatch between detection and reality made the system unusable and prompted a re-evaluation of the entire infrastructure architecture. The most important service differentiating our product is the Guardian; without it, we cannot push MCP or the narrative engine.

Guardian V1

The failure of the initial implementation revealed a key limitation: narrative analysis cannot rely on a single method. Rule-based systems provide precision but lack interpretative depth, while model-driven approaches introduce variability. Guardian V1 addresses this by combining multiple sources of analysis into a unified system.

At its core, the architecture comprises three components: a knowledge-based entity registry, deterministic surface-level classifiers, and semantic reasoning via language models. These correspond to three modes of validation:

Truth (KB) -> Pattern (Rules) -> Reasoning (LLM)

Each layer targets a different class of problem. The knowledge base enforces consistency against defined entities and timelines. The rule-based layer captures deviations expressed through explicit textual patterns, such as point-of-view shifts or temporal inconsistencies. The semantic layer evaluates higher-level properties, including coherence, tone, and narrative quality.

This structure is implemented as a staged pipeline. The entity registry first validates the text against the knowledge base. Deterministic classifiers then analyse surface-level patterns with high precision. Structural analysis extends this to event-level coherence and progression. Finally, a semantic layer applies model-based reasoning to capture aspects that cannot be reduced to rules.

The system is inherently language-dependent. Narrative signals such as name recognition and dialogue structure differ across languages, requiring separate parsing strategies for English and Chinese to maintain accuracy. The main system was strongly inspired by Chinese-novelist-skill [2]

Guardian V1 should therefore be understood not as a single classifier, but as a layered verification system. Its role is to continuously ensure that an evolving narrative remains logically consistent and structurally coherent, without interfering with the author's creative process.

Benchmarking

To evaluate Guardian, the objective is to assess its ability to detect and analyse narrative consistency, structure, and literary quality across different text forms. This requires a benchmark with well-defined narrative structures while preserving the complexity of real-world writing.

Canonical literature was selected as the evaluation domain for this purpose. Such works exhibit established narrative coherence, character development, and structural integrity, making them suitable as reference points for both rule-based and model-based analysis. Two texts were chosen: Hamlet for English and Dream of the Red Chamber for Chinese, allowing evaluation across both language and stylistic variation.

To control for differences in textual structure, two types of input were used. The first, referred to as fixture text, consists of structured and annotated content in which key narrative signals are explicitly expressed. The second, original text, preserves the full literary form, where narrative elements are embedded implicitly within the prose. This distinction enables comparison between idealised and real-world conditions.

Evaluation was conducted across multiple analysis modes, including local rule-based processing and several language models, such as GPT-4o-mini, GPT-5.3, Claude Sonnet, Claude Opus, and DeepSeek. This setup allows the system to be examined not as a single model-dependent solution, but as a hybrid architecture operating under different reasoning regimes.

Analyser Structure

The evaluation framework consists of five layers. The first two layers are fully rule-based, while the remaining layers combine deterministic methods with language model reasoning.

Layer 1 — Consistency Validation

This layer evaluates internal consistency within the narrative, including entity attributes, character state, and timeline alignment.

The analysis is performed using deterministic rules applied to structured inputs, without the use of language models. Inputs are controlled to minimise ambiguity, ensuring that inconsistencies can be evaluated without interpretative variance.

Performance is measured using precision, recall, and false positive rate.

All configurations achieve perfect scores in this layer. This reflects the task's deterministic nature, in which explicitly defined constraints can be validated reliably without semantic reasoning.

Layer 2 — Style Analysis

This layer evaluates stylistic properties of the text, including sentence variation, paragraph structure, modifier usage, narrative exposition, and background density.

The analysis is performed using deterministic classifiers that capture surface-level linguistic patterns, without relying on language models.

Performance is measured by execution sanity and resistance to false positives.

All configurations achieve full scores in this layer, indicating that the classifiers operate reliably and do not introduce spurious detections.

A consistent difference is observed between the fixture and the original texts. Original texts exhibit higher sentence variation and more complex syntactic structures. Extended paragraph formations appear predominantly in original texts, particularly in Dream of the Red Chamber, reflecting a preference for longer narrative blocks. Background density is also higher in original texts, as seen in Hamlet, where descriptive context is embedded within dialogue.

These results indicate that while stylistic patterns can be reliably detected using deterministic methods, their distributions differ significantly between structured and natural literary forms.

Layer 3 — Quality Analysis

This layer evaluates higher-level literary properties of the text, including perceived quality, character arc progression, causal coherence, and the absence of structural flaws.

The analysis is performed using a hybrid approach. Deterministic methods rely on explicit evidence matching within the text, while language models provide semantic interpretation to track narrative progression and assess qualitative attributes.

Performance is evaluated across four dimensions: detection quality, arc mapping, causal coverage, and absence of flaws.

Results show that deterministic methods achieve perfect scores in quality detection and causal coverage, reflecting their effectiveness in matching explicit textual signals. In contrast, model-based evaluation exhibits significant variation, particularly in arc mapping.

A notable divergence is observed in Dream of the Red Chamber. Claude Opus fails to detect any character arc beats on the Chinese original text, reflecting a strict matching policy that rejects paraphrased or semantically equivalent expressions. GPT-4o-mini and GPT-5.3 achieve moderate performance by allowing greater semantic flexibility, but still exhibit errors in arc ordering and incomplete detection. DeepSeek achieves full coverage, likely due to stronger alignment with Chinese literary data.

These results indicate that arc mapping is not purely a reasoning task, but a function of linguistic alignment, narrative state tracking, and matching tolerance. While deterministic methods ensure precision in explicit cases, model-based approaches introduce variability depending on how strictly semantic equivalence is interpreted.

Layer 4 — Suspense Analysis

This layer evaluates narrative tension at the chapter level, including the presence of suspense and the activation of narrative threads across character, relationship, temporal, and main plot dimensions.

The analysis combines deterministic classification with model-based interpretation. Rule-based methods detect structural signals of suspense, while language models are used to resolve ambiguity and reduce false positives in dialogue-heavy or stylistically complex passages.

Performance is evaluated based on classifier execution reliability, suspense-detection coverage, narrative thread activation, and the absence of false cliffhanger signals.

Results indicate that deterministic methods tend to over-detect suspense in texts with dense interrogative or rhetorical structures. This is particularly evident in Hamlet, where question-driven dialogue leads to multiple false cliffhanger detections. Model-based evaluation significantly improves this behaviour by filtering out spurious signals.

Among the evaluated models, Claude Opus demonstrates the strongest performance, achieving complete removal of false positives across all test cases. Other models reduce false detections to varying degrees but remain less consistent.

These results suggest that suspense detection cannot rely solely on surface patterns. Accurate classification requires interpretative judgment to distinguish between genuine narrative tension and stylistic expression.

Layer 5 — Structural Analysis

This layer evaluates macro-level narrative structure, including act segmentation, key turning points, and long-range narrative alignment.

The analysis relies primarily on model-based reasoning, as these tasks require understanding relationships across extended sections of text.

Performance is measured through structural detection accuracy and execution reliability.

Results show that language models consistently outperform rule-based methods. On fixture texts, LLM-based evaluation scores 95 on average compared to 82 for local methods. On original texts, this gap persists at 76 versus 66. Structural elements such as midpoints and foreshadowing are often implicit and cannot be captured through surface patterns alone.

This indicates that macro-level narrative structure is fundamentally a semantic problem, requiring interpretative reasoning rather than deterministic matching.

Fixture vs Original Text

Fixture and original texts exhibit complementary characteristics in evaluation. Fixture texts contain explicitly structured signals, which reduce parsing complexity and improve detection consistency for elements such as character arcs, suspense, and structural transitions. In contrast, original texts preserve the full narrative context, in which these signals are embedded implicitly within the prose. This enables more accurate modelling of causal relationships and cross-chapter dependencies, but increases the difficulty of detection.

As a result, fixture-based evaluation yields higher average detection performance, while original texts provide a more realistic assessment of system capability in natural writing conditions.

Model Comparison

Across all evaluations, Claude Opus demonstrates the strongest overall performance, with a clear margin over other models. However, performance varies by domain. DeepSeek shows stronger alignment with Chinese texts, while GPT-based models perform more consistently in long-range structural analysis.

These results indicate that model performance is not uniform but depends on linguistic alignment and task characteristics, reinforcing the need for a hybrid system rather than relying on a single model.

Conclusion

The benchmark suggests that no single model is sufficient across all aspects of narrative analysis. Although Claude Opus demonstrates the strongest overall performance, it is not uniformly reliable, particularly in tasks such as character arc mapping, where stricter matching behaviour can reduce recall. Other models perform better under specific conditions, such as Chinese literary analysis or long-range structural interpretation.

At the same time, the results show that rule-based methods remain highly effective in tasks defined by explicit constraints. Deterministic classifiers achieve strong performance in consistency validation and surface-level style analysis, where precision matters more than interpretation. However, they are less effective in tasks such as plot structure and false cliffhanger detection, where the system must distinguish between surface signals and narrative meaning.

This suggests that narrative analysis should not be framed as a choice between rules and language models. The more effective approach is a layered system in which deterministic methods provide precision and stability, while language models handle ambiguity, context, and long-range semantic structure.

The benchmark also highlights the importance of multilingual evaluation. While performance remains relatively stable on Hamlet, greater variation appears on Dream of the Red Chamber, indicating that linguistic and cultural alignment remain significant factors in model behaviour.

Guardian V1 is therefore best understood as an initial verification framework rather than a finished solution. Its significance lies not only in classifier performance but also in establishing a foundation for narrative systems that can move from detection toward suggestion, guidance, and, eventually, deeper forms of writing support.

Guardian should be understood not as a model, but as a verification layer for narrative systems. In this sense, narrative systems require verification layers in the same way software systems require compilers.

What's next?

• Adding more models for benchmarking.

• Improve the classifier with more rules

• Seek more language support

Citation

[1] CJK refers to Chinese, Japanese, and Korean

[2] Chinese-novelist-skill - https://github.com/PenglongHuang/chinese-novelist-skill

Appendix