Methodology v0.2

How MartingaleBench works

The goal is narrow and honest: detect when a model API behaves differently from itself. We don’t rank which model is “smartest” — we tell you when yours changed.

What we test

Every model gets the same starter suite of short canary cases. Prompts are generated from templates with per-run random variables and never mention Martingale, so they look like ordinary traffic.

How we score

• Deterministic only. No LLM-as-judge: a judge model can drift too, which would hide the very thing we’re measuring.
• Repeated sampling. 5 samples per case per run, about 7.8 runs per day. Temperature 0 where the API allows it (reasoning models reject it and run at their default). Long context runs every 6 hours, hard categories every 4 hours.
• Hard probes (★). Code execution, constraint logic and state tracking, each at 5 difficulty levels with a fixed 2,500-token output budget that includes hidden reasoning. Answers are computed by running the program or solving the puzzle, never judged. A provider quietly cutting reasoning effort shows up here first.
• Claude models currently run through the Claude Code CLI (no tool or max-tokens cases there, default thinking on), so their scores are not directly comparable with API-called models. Drift is still measured against each model’s own history.
• API defaults. Every model is called the way its vendor ships it. Some reason before answering by default and some do not unless you turn that on, so the hard probes are not a level playing field between vendors. That is deliberate: we track each model against its own past, not against the others.
• Format counts. “Arithmetic, answer only” fails a reply that reaches the right number but writes its working first. It measures following the instruction as well as the sum.
• Errors are separate. Rate limits and 5xx responses are excluded from quality scores and tracked as availability.
• Overall score is the unweighted mean of the capability pass rates over the last 24 hours. Trend lines use the core categories only, so adding new categories never looks like drift.

How we call drift

A change only becomes an incident when it is statistically significant, large enough to matter, and persistent — the main defence against false alarms.

Incidents carry a confidence label: High (adjusted p < 0.0001 and a move of at least 20 points, or an identity change), Medium (adjusted p < 0.001), or Low. While a model has under 48 hours of history, confidence is capped at Medium.

Raven Alpha: changes that scores can’t see

Raven Alpha is our behaviour-fingerprint detector. Pass rates only move when a change is big enough to flip an answer. Most silent changes are not: a cheaper sibling model, a new quantisation, less reasoning effort or a hidden system prompt can leave every canary passing. So each run also asks 28 fixed prompts with no right answer (“Name one color”, “Write one sentence about rain”) at the API’s default sampling (direct Anthropic references are the exception: the runner and martingale check both call that API at temperature 0), and tracks the distribution of what comes back: the answers, early word choice, output length, reasoning-token counts and, where the API exposes them, first-token probabilities.

Two look-back windows are compared with the model’s own earlier runs: the latest 3 runs, to catch a wholesale change within hours, and the latest 24 runs, to catch a partial change that is too faint in 3. Each comparison is an exact permutation test over whole runs (no distributional assumptions), corrected for the two windows and across endpoints. Two significant checks in a row raise a flag; a flag becomes an incident only once the baseline covers 24 hours. A fixed prompt whose input-token count changes is flagged immediately.

Output budget. Each fixed prompt may use 400 output tokens, hidden reasoning included. Models that spend most of that on reasoning returned an empty answer on a large share of calls, which fingerprints the budget rather than the model, so since 6 Oct 2026 these endpoints run with a larger budget and their fingerprint series restarted on that day: stepfun/step-3.7-flash (4,000), bytedance-seed/seed-2-1-turbo (4,000), inception/mercury-2.5 (2,000), google/gemini-3.8-flash (2,000), ibm-granite/granite-4.2-8b (2,000), moonshotai/kimi-k2.6 (2,000), tencent/hy3 (2,000), z-ai/glm-5.3 (2,000), thinkingmachines/inkling (2,000), nvidia/nemotron-3.5-lightning (2,000), cohere/command-a-plus (2,000), minimax/minimax-m3 (2,000). Earlier responses from them stay in the raw data but are not compared with the new ones. A few models keep reasoning past any budget on trivial prompts (StepFun 3.7 Flash on about one call in ten); the share of cut responses per model is published with the series.

Measured sensitivity (blind test on real responses, 05 Oct 2026). We collected real responses from pairs of closely related models. In each trial a sealed coin decided whether the recent window came from the same model or had a share of its calls answered by the sibling; the detector saw only anonymised responses. 300 trials per cell, one check, threshold 0.01.

False alarms when nothing changed: 0.8% (25 of 3300) and 1.0% (34 of 3300) per check, before the two-in-a-row rule. Caveats: the test responses were collected in one session, so false alarms over days are not measured yet; a 10% partial swap is mostly invisible at this sample size; and this measures sensitivity to swaps we staged between real models, not a provider change caught in the wild.

With more history. The table above uses a baseline of only 16 batches, because that is all one collection session gives. A live endpoint builds a far longer baseline. To estimate what that buys, we rebuilt longer baselines and windows by drawing at random from the same real responses (72 baseline batches, about three days of hourly runs). This is an estimate from resampled data, not a measurement over three real days.

False alarms in the same resampled setting: 1.2%, 1.1% per check.

Limitations

• Canaries are narrow probes, not a measure of general intelligence. A flat line doesn’t mean a model hasn’t changed in ways we don’t test.
• We call each provider’s public API from our own accounts. Your region, tier, prompts and traffic may behave differently — that’s what monitoring your own endpoints is for.
• Scores are pass rates over finite samples, so small differences (a point or two) are noise. Look for step changes.
• Providers may route or update models without notice; we report what we observe, not why.

Open data

The full results behind every chart are published as results.json. Suite hard-v2 + starter-v2; data generated Sat, 07 Nov 2026 02:24:33 GMT.