Methodology

The Known Good Index is a composite of six published evaluations. It is computed like this, and only like this:

1. Normalise. For each evaluation, every model holding a score is min-max normalised to 0–1 against every other model holding that same evaluation. The best tracked score becomes 1, the worst becomes 0. Where an evaluation is scored lower-is-better, the scale is inverted first. Where every holder has an identical score — or there is only one holder — everyone gets 0.5, because with no spread there is no evidence for a ranking and awarding a lone data point 1.0 would put it at the top of the site.
2. Average with equal weight. The normalised values a model actually has are averaged. No evaluation counts for more than another. There is no hand-tuning, no vendor input, and no weighting that we adjust when we dislike a result.
3. Scale to 0–100. The average is multiplied by 100 and rounded to two decimals.
4. Require a score in every category. A model qualifies only if it holds at least one science score (GPQA Diamond), at least one mathematics score (MATH / AIME) and at least one coding score (SWE-bench Verified, LiveBench or Terminal-Bench). A model missing any one of those three gets no index score at all — not a low one. This is a balance rule, not a counting rule: four results clustered in one area used to be enough to qualify, which let a headline number imply general capability the evidence did not support. Humanity's Last Exam contributes to the average but belongs to no category, so it can never by itself make a model eligible. The number of components behind each score is recorded, shown beside every index value as based on N of 6 evaluations, and published in the CSV as index_components.

Two consequences worth understanding. First, the index is relative: a model's score depends on the population we track, so scores move when new models are ingested even if nothing about that model changed. Second, it is ours. It is on a 0–100 scale of our own construction and we never restate another publisher's headline number as if it were the index.

Each capability index is built exactly as the Known Good Index is — min-max normalise each component, average with equal weight, scale to 0–100 — but over a subset:

• Coding Index — SWE-bench Verified, LiveBench coding, Terminal-Bench. At least two of the three required.
• Agentic Index — Terminal-Bench and SWE-bench Verified.
• Mathematics Index — MATH / AIME and MATH Level 5.
• Science Index — GPQA Diamond.
• Reasoning Index — Humanity's Last Exam, LiveBench reasoning, GPQA Diamond. At least two of the three required.

Where a subset has three or more possible components we require at least two, so that a single evaluation can never masquerade as a composite. Where a subset is a single evaluation — science is GPQA Diamond and nothing else — the index is that evaluation, normalised, and we say so rather than dressing it up.

Openness is not a benchmark and we do not pretend to measure it with one. It is computed from published metadata about weights and licensing:

• 100 — open weights under a permissive licence.
• 90 — open weights under a permissive licence, but with no retrievable repository recorded. Weights you cannot actually fetch are less open than weights you can.
• 60 — open weights with commercial use restricted.
• 40 — open weights under a non-commercial licence.
• 5 — proprietary weights.

Because it needs no evaluations, an openness score exists for models that hold no benchmark results at all. It says nothing about quality and is never mixed into the Known Good Index.

Blended price is (3 × input + output) ÷ 4 — USD per 1M tokens at a 3:1 input:output ratio, which is roughly what ordinary chat and retrieval traffic looks like. It is computed from published rates, so you can reproduce every figure on this site with a calculator.

Rates are ingested from OpenRouter and cross-checked against models.dev. Where a model is served by several hosts at different prices, the model page shows the canonical rate and the provider table shows each host separately. A missing price renders as an em dash — never as zero.

Why we do not publish cost per task

Cost per task is the number everyone wants and the number we refuse to invent. Producing it honestly requires token counts from running a full evaluation suite — input tokens, reasoning tokens and output tokens per task — and we do not run evaluation suites. We would have to guess the token counts, and a guess multiplied by a real price looks exactly like a measurement. So we publish blended price, state the blend, and leave the multiplication to you (DESIGN-LOCK §7.7).

Reasoning models are the sharpest case: a model that emits ten times the tokens to answer the same question can be cheaper per token and far more expensive per task. Blended price will not tell you that. Nothing we could publish without running the evaluations would.

Output speed is the median output tokens per second across every tracked provider serving that model. Latency (TTFT) is the median seconds to the first streamed token. Both are derived from OpenRouter endpoint telemetry — we do not run our own timing harness, and we say so rather than implying a lab.

Speed is a property of the host at least as much as of the model. The same weights on two hosts can differ severalfold, which is why the provider tables exist and why a single headline speed figure should be treated as an indication, not a guarantee. Numbers are also snapshots: they move with load, region and time of day.

Elo ratings come from blind pairwise votes: a person sees two anonymous outputs for the same prompt and picks one. The votes are collected and published by Arena, and by TTS Arena for speech. We ingest the published ratings and confidence intervals. We do not run the votes.

Elo is not part of the Known Good Index. It measures which answer a person preferred, which correlates with — but is not — correctness, and it is sensitive to formatting, length and style. It earns its own section and stays out of the composite.

Every source is ingested on a schedule into a staging table, validated record by record against a schema, and sanity-checked against the last successful run. Only then is it swapped into the live tables inside a single transaction. A run that returns nothing, or far fewer rows than last time, is marked rejected and the live tables are left exactly as they were.

That is why a page here can be empty but is never wrong: if an upstream feed breaks, you see the last good data or an honest gap, not a table of zeros. The last run of every source is published on the about page.

Matching models across sources

Sources disagree about names. We match on the provider's model identifier first, then on a normalised name — lowercased, punctuation stripped, parenthetical suffixes removed. A name that matches nothing becomes a new model rather than being merged into an existing one, because silently merging two distinct models corrupts every number attached to both. Unmatched names are logged for human review.

OpenRouter leads on freshness and often has a model within hours; models.dev enriches metadata but never blocks a model from appearing. A model with no metadata yet shows dashes, not guesses.

• We do not run evaluations. Not one number on this site was measured by us. Every chart names its source and every score is traceable to a publisher.
• We do not accept vendor input on weights or rankings. The index is equal-weighted precisely so there is nothing to lobby for.
• We do not publish cost per task, for the reason above.
• We do not rank a model we cannot score. Missing science, mathematics or coding evidence means no index score — a model is never ranked on breadth in one area alone.
• We do not fill gaps. A missing figure is an em dash. We never interpolate, estimate or carry a stale value forward as if it were current.
• We do not run user accounts. There is a newsletter and nothing else to sign up for.

Everything above is checkable: the same figures are downloadable as CSV and JSON, and every publisher and licence is listed on attribution.