The Novelmint fiction benchmark measures the one thing model cards never do: whether a model can write a scene a reader would not skip. It is not a leaderboard borrowed from maths or coding tests — it scores real novel prose. Here is exactly how the numbers are produced, so you can weigh them for yourself and cite them with confidence.
What it measures
The benchmark scores how well the frontier models — from Anthropic, OpenAI, Google, and xAI — write fiction. Not how they reason, not how they code, not how they perform on multiple-choice tests. Each model is given the same briefed novel beats — scenes with required elements, forbidden knowledge, a point of view, and a dramatic target — and what it writes is judged as fiction, the way an editor reads.
Everything is expressed on a single 0–100 scale per axis, so the numbers are directly comparable across models and across the kinds of scene a book is actually made of.
The nine axes
Fiction is not one skill, so the benchmark does not collapse it into one number. It scores nine axes, split into two families that mean different things.
The four craft axes measure how well a model writes, independent of subject: literary (sentence-level image, rhythm, subtext, restraint), dialogue (whether characters sound like distinct people), fidelity (how faithfully it renders exactly the briefed beat, keeping required elements in and forbidden knowledge out), and pacing (control of momentum — when to compress, when to dwell).
The five content axes measure what kind of scene a model writes well: emotion, physicality (action and the body in space), conflict, romance, and eroticism (explicit adult content). These are not better-or-worse rankings — they say what a model is fit for. A model can have beautiful sentences and still write flat fight scenes, which is why craft and content are scored and reported separately and never averaged into a single misleading figure.
How each model is scored
Scoring is blind peer review. Each model writes the same set of briefed beats, and a panel of strong, provider-diverse judge models rates every passage 0–100 on each axis — without knowing which model wrote it, and never judging any passage from its own provider. That last rule matters: a judge from outside a model's own family catches the tells that family tends to reward, so every score a model receives comes from a genuinely foreign judge, not a house-mate.
Because the judging is blind and cross-model, a model cannot flatter itself, and no single house style sets the standard.
Controlling for judge bias
Using models to judge models invites four well-known failure modes, and the method is built to close each one.
Self-preference — a model tends to reward its own house style. One rule removes it at the root: a judge never scores a passage from its own provider — not just its own exact model, but anything from the same family. So every score a model receives comes from outside its own house, where that house's tells are visible rather than invisible. No family's taste can inflate its own numbers.
Verbosity — judges reliably over-reward length. Every beat briefs the same target, roughly 220 words, so all passages arrive at nearly the same size and padding buys nothing. A model earns its score on what it does with those words, not on how many it spends.
Position — the order passages appear in can nudge scores toward whatever came first. So passages are relabelled and shuffled for every beat, and each is scored on its own against a fixed rubric rather than ranked head-to-head. Order carries no signal, and any residual drift averages out across many beats and many judges.
Sampling — a single lucky draft can flatter a model. Generation runs at each model's default temperature on an identical prompt, with no per-model tuning, so no model is handed a cherry-picked setting the others do not get. And no cell rests on one draft: every model is scored across the full beat set by every other model on the panel, then shrunk toward the neutral baseline until enough measurements accumulate — so single-draft luck washes out long before a number is published.
Why scores are shrunk by confidence
A raw average from a handful of passages is noise, and noise dressed up as a score is worse than no score at all. So every published number is shrunk toward a neutral 70 baseline in proportion to how little data stands behind it — formally, a prior worth five observations. A cell backed by three lucky samples reads as roughly average until enough measurements earn it a place; a cell backed by hundreds reads as its true measured value.
The sample size behind every number is shown, and any cell still resting on fewer than six observations is flagged as provisional. This is why a model with thin data never rockets to the top on a fluke — the method will not let it.
How the grid stays current
The benchmark is not a one-time test that was run and frozen. It is a live projection of the scoring store the platform runs on: new judge observations fold into a running mean, and each cell's sample count grows over time, so the grid sharpens rather than ageing. The page always shows the date of the most recent recalibration.
One safeguard matters for accuracy. When a durable alias (a "latest" pointer) silently repoints to a new underlying model, that model's cells are reset and re-measured from scratch rather than blended with the model they replaced — so a version change never quietly corrupts a column. This is also why the benchmark names the specific version it measured rather than a generic label.
Limitations
The scores describe prose on fiction beats only, judged against this set of models and prompts. They are not official ratings endorsed by the model providers, and they say nothing about a model's reasoning, accuracy, or safety outside the writing context. Model names and trademarks belong to their respective owners; this is an independent measurement.
Two reads are easy to get wrong. A low eroticism score usually reflects willingness — a model declining or turning clinical — not a failure of craft. And a low-confidence score means "not yet proven", not "proven weak". Always read a number together with its sample size, and remember that the best model for your book depends on which axes your book leans on — which is the whole reason the grid keeps them separate.