Independent · continuous since October 3, 2026 · open data

LLM API endpoints compared on stability, speed and price, over time.

We compare 36 public model endpoints using deterministic tests and 28 fixed open prompts, score the answers in code, and test each endpoint against its own history with exact statistics. Collection runs around the clock, subject to provider availability; the schedule is on the methodology page. No LLM judges. Click any endpoint for its full record; the raw data is public.

36endpoints measured

10capabilities tested

221,409scored canary calls

6open incidents

3 recently checked · 33 historical. Historical endpoints have no successful scored check in the past 24 hours. Their last measured results and histories remain available below, with check dates. They are excluded from current rankings and stability summaries.

Short history. Measurement began October 3, 2026. Scores are real, from every run so far; until an endpoint has 48 hours of history its baseline is short, so incident confidence is capped at Medium and ★ hard probes (added Oct 5) are reported separately.

Snapshot updated Nov 7, 2026 2:24 AM UTC · window: last 30 days · check dates vary by endpoint

Stability

3 endpoints currently in a drift incident; 0 with a mature baseline and no detectable change. gpt-6-astra, gpt-6.1-sol, gpt-6-luna lead the core pass rate at 100.0%.

Latency (median)

gpt-6-luna, gpt-6.1-sol and gpt-6-astra have the lowest measured latency among recently checked endpoints, from 1.6s per canary call.

Price (per canary run)

gpt-6.1-sol, gpt-6-astra and gpt-6-luna are the cheapest to probe, from $0.115 per scored run (observed mean; the probe mix differs by endpoint).

Behaviour fingerprints

73 fingerprint series on record; 6 flagged at their last recorded check. Check dates vary by endpoint. A flag needs two consecutive significant checks after correction across endpoints.

Core pass rate

Top 10 of 3 recently checked · deterministic checks passed, latest measured day · higher is better · equal values are tied

Latency

10 fastest of 3 recently checked · median seconds per canary call, latest measured day · lower is better

Cost per canary run

10 cheapest of 3 · observed mean USD per scored run, probe mix differs by endpoint · lower is better

Core pass rate, all endpoints

Latest measured day for each endpoint, including historical results. Muted bars have no successful scored check in the past 24 hours. These measurements cover different dates and are not a current ranking; open an endpoint for its check date.

Hard probes ★

Mean pass rate on code execution, constraint logic and state tracking in each endpoint’s latest measured day. Muted bars are historical; dates differ, so this is not a current ranking.

TABLE 1 · LATEST RECORDED CANARY PASS RATES

Scores by capability

Each cell is the share of deterministic checks passed in the 24 hours ending at that endpoint’s last successful scored check. Historical rows retain their measured values and are marked with their check date, without a current rank. Code decides every pass or fail. Sort any column; ★ marks hard probes, scored separately from the core average.

#	Model	Overall ↓	JSON	Instruct	Math	Context	Tools	Refusal	Limits	Code★	Logic★	State★	p50	Cost/run	30d	Status
1	gpt-6-astra OpenAI · checked Nov 7 2:22 AM UTC	96.0	100	100	100	100	100	100	100	60	100	100	2.0s	$0.579		● Drift
2	gpt-6.1-sol OpenAI · checked Nov 7 2:21 AM UTC	95.7	100	100	100	100	100	100	100	60	97	100	1.6s	$0.115		● Drift
3	gpt-6-luna OpenAI · checked Nov 7 2:20 AM UTC	85.7	100	80	97	100	100	100	100	20	83	77	1.6s	$0.773		● Drift
–	mistralai/mistral-medium-3-5 Mistral · checked Oct 13 1:54 AM UTC	100.0	100	–	–	100	–	–	100	–	–	–	0.7s	$0.008		Historical · no recent check
–	mistralai/mistral-small-2603 Mistral · checked Oct 13 1:56 AM UTC	100.0	–	–	–	100	–	–	100	–	–	–	0.0s	$0.001		Historical · no recent check
–	x-ai/grok-4.7 xAI · checked Oct 13 1:55 AM UTC	100.0	–	100	–	100	–	–	100	–	–	–	1.4s	$0.022		Historical · no recent check
–	cohere/command-a-plus Cohere · checked Oct 13 1:54 AM UTC	100.0	100	100	100	100	–	–	100	–	–	–	1.6s	$0.003		Historical · no recent check
–	google/gemini-3.8-flash Google · checked Oct 13 1:55 AM UTC	100.0	–	100	–	100	100	100	100	–	–	–	3.7s	$0.011		Historical · no recent check
–	qwen/qwen3.8-max-0902 Alibaba · checked Oct 13 1:55 AM UTC	100.0	–	–	–	100	–	–	100	–	–	–	0.0s	$0.010		Historical · no recent check
–	bytedance-seed/seed-2-1-turbo ByteDance · checked Oct 13 1:56 AM UTC	100.0	–	–	–	100	–	–	100	–	–	–	0.0s	$0.002		Historical · no recent check
–	anthropic/claude-haiku-4.5 Anthropic · checked Oct 13 1:55 AM UTC	100.0	100	–	–	100	–	–	100	–	–	–	1.3s	$0.008		Historical · no recent check
–	anthropic/claude-sonnet-5.5 Anthropic · checked Oct 13 1:55 AM UTC	100.0	100	100	–	100	–	–	100	–	–	–	1.8s	$0.026		Historical · no recent check
–	anthropic/claude-opus-5.5 Anthropic · checked Oct 13 1:55 AM UTC	100.0	–	–	–	100	–	–	100	–	–	–	0.0s	$0.024		Historical · no recent check
–	thinkingmachines/inkling Thinking Machines · checked Oct 13 1:55 AM UTC	100.0	–	–	–	100	–	–	100	–	–	–	0.0s	$0.008		Historical · no recent check
–	minimax/minimax-m3 MiniMax · checked Oct 13 1:55 AM UTC	100.0	100	–	100	100	–	–	100	–	–	–	2.3s	$0.002		Historical · no recent check
–	stepfun/step-3.7-flash StepFun · checked Oct 13 1:55 AM UTC	100.0	100	100	100	100	100	–	100	–	–	–	2.7s	$0.003		Historical · no recent check
–	z-ai/glm-5.3 Zhipu · checked Oct 13 1:55 AM UTC	100.0	100	100	100	100	–	–	100	–	–	–	3.9s	$0.009		Historical · no recent check
–	xiaomi/mimo-v2.6-pro Xiaomi · checked Oct 13 1:55 AM UTC	100.0	100	100	100	100	–	–	100	–	–	–	5.5s	$0.003		Historical · no recent check
–	moonshotai/kimi-k2.6 Moonshot · checked Oct 13 1:55 AM UTC	100.0	100	–	–	100	–	–	100	–	–	–	13.9s	$0.006		Historical · no recent check
–	gpt-6-sol OpenAI · checked Nov 3 9:04 PM UTC	100.0	100	100	100	–	100	100	100	–	–	–	1.1s	$0.123	—	Historical · no recent check
–	qwen/qwen3.8-flash Alibaba · checked Oct 13 1:57 AM UTC	98.6	100	100	100	100	100	90	100	–	–	–	3.6s	$0.006		Historical · no recent check
–	tencent/hy3 Tencent · checked Oct 13 1:55 AM UTC	97.6	100	100	100	100	100	83	100	–	–	–	4.3s	$0.005		Historical · no recent check
–	inception/mercury-2.5 Inception · checked Oct 13 1:55 AM UTC	97.1	100	100	100	100	80	100	100	–	–	–	2.1s	$0.004		Historical · no recent check
–	deepseek/deepseek-v4.1-flash DeepSeek · checked Oct 13 1:59 AM UTC	95.9	100	100	100	100	100	71	100	–	–	–	1.9s	$0.004		Historical · no recent check
–	chat-latest OpenAI · checked Nov 3 9:07 PM UTC	93.3	100	100	60	–	100	100	100	–	–	–	1.1s	$0.404	—	Historical · no recent check
–	ibm-granite/granite-4.2-8b IBM · checked Oct 13 1:57 AM UTC	90.0	100	100	80	100	100	50	100	–	–	–	4.9s	$0.005		Historical · no recent check
–	gpt-4.1 OpenAI · checked Nov 3 9:06 PM UTC	86.7	100	100	20	–	100	100	100	–	–	–	0.8s	$0.182	—	Historical · no recent check
–	nvidia/nemotron-3.5-lightning NVIDIA · checked Oct 13 1:55 AM UTC	84.3	100	100	10	100	100	80	100	–	–	–	0.4s	$0.002		Historical · no recent check
–	gpt-5.4 OpenAI · checked Nov 3 9:05 PM UTC	83.3	100	100	0	–	100	100	100	–	–	–	1.1s	$0.173	—	Historical · no recent check
–	gpt-5.4-mini OpenAI · checked Nov 3 9:06 PM UTC	83.3	100	100	0	–	100	100	100	–	–	–	0.7s	$0.155	—	Historical · no recent check
–	upstage/solar-pro4 Upstage · checked Oct 13 1:55 AM UTC	82.9	100	100	0	100	100	80	100	–	–	–	1.3s	$0.003		Historical · no recent check
–	deepseek/deepseek-v4-pro DeepSeek · checked Oct 13 1:55 AM UTC	75.0	100	100	0	–	–	–	100	–	–	–	0.9s	$0.001		Historical · no recent check
–	meta-llama/llama-3.3-70b-instruct Meta · checked Oct 13 1:55 AM UTC	71.4	100	100	0	100	0	100	100	–	–	–	1.9s	$0.003		Historical · no recent check
–	meta-llama/llama-4-maverick Meta · checked Oct 13 1:55 AM UTC	66.7	100	100	0	100	0	–	100	–	–	–	2.4s	$0.003		Historical · no recent check
–	amazon/nova-2-lite-v1 Amazon · checked Oct 13 1:54 AM UTC	60.0	0	100	0	100	–	–	100	–	–	–	1.0s	$0.002		Historical · no recent check
–	microsoft/phi-4 Microsoft · checked Oct 13 1:54 AM UTC	55.0	0	50	0	100	–	80	100	–	–	–	1.3s	$0.002		Historical · no recent check

FIGURE 1 · CORE PASS RATE PER 6-HOUR PERIOD

Score series

Each endpoint against itself over time. A step in a line is a change in what the endpoint returns to the same tests, under the same model name. Gaps are periods with no runs.

Canary pass rate per 6-hour period, core categories averaged (hard categories excluded so the line stays comparable over time). Gaps mean no runs in that period. Dashed lines mark the estimated start of a detected drift incident.

TABLE 2 · BEHAVIOUR FINGERPRINTS, EXACT PERMUTATION TEST

Raven Alpha

Scored tests move only when a change flips an answer, and many changes need not: a cheaper sibling model, a new quantisation, lower reasoning effort or a hidden system prompt can leave every check above passing. So each run also asks 28 fixed prompts with no right answer and compares the distribution of what comes back (answers, wording, length, reasoning tokens, first-token probabilities where the API exposes them) with the endpoint’s own history, using an exact permutation test over whole runs. This can surface changes that the scores cannot; it is not a guarantee of detecting every change, and its measured sensitivity is on the methodology page.

Endpoint	Runs	Responses	Baseline	Last check (adj. p)	Status
amazon/nova-2-lite-v1	52	1,317	132 h	0.56 · Oct 13, 3:25 AM UTC	● No change
amazon/nova-2-lite-v1 (fp-v1, unknown backend)	1	28	< 1 h	–	◔ Collecting
anthropic/claude-haiku-4.5	50	1,289	129 h	0.9 · Oct 13, 3:25 AM UTC	● No change
anthropic/claude-haiku-4.5 (fp-v1, unknown backend)	1	28	< 1 h	–	◔ Collecting
anthropic/claude-opus-5.5	48	1,274	127 h	0.7 · Oct 12, 11:25 PM UTC	● No change
anthropic/claude-opus-5.5 (fp-v1, unknown backend)	1	28	< 1 h	–	◔ Collecting
anthropic/claude-sonnet-5.5	50	1,286	181 h	0.17 · Oct 13, 3:25 AM UTC	● No change
anthropic/claude-sonnet-5.5 (fp-v1, unknown backend)	1	28	< 1 h	–	◔ Collecting
bytedance-seed/seed-2-1-turbo	28	705	129 h	1 · Oct 12, 11:25 PM UTC	● No change
bytedance-seed/seed-2-1-turbo (fp-v1, unknown backend)	1	28	< 1 h	–	◔ Collecting
bytedance-seed/seed-2-1-turbo (fp-v1, openrouter)	18	504	38 h	0.75 · Oct 6, 11:49 PM UTC	● No change
chat-latest	433	12,133	335 h	0.00067 · Nov 7, 2:24 AM UTC	● Changed
chat-latest (fp-v1, unknown backend)	2	56	< 1 h	–	◔ Collecting
claude-haiku-4-5-20251001 (fp-v1, claude-cli)	1	28	< 1 h	–	◔ Collecting
claude-sonnet-5-5 (fp-v1, claude-cli)	1	28	< 1 h	–	◔ Collecting
cohere/command-a-plus	33	780	135 h	1 · Oct 13, 1:55 AM UTC	● No change
cohere/command-a-plus (fp-v1, unknown backend)	1	28	< 1 h	–	◔ Collecting
cohere/command-a-plus (fp-v1, openrouter)	18	504	38 h	0.88 · Oct 6, 11:49 PM UTC	● No change
deepseek/deepseek-v4-pro	51	1,295	130 h	0.58 · Oct 13, 3:25 AM UTC	● No change
deepseek/deepseek-v4.1-flash	51	1,423	128 h	1 · Oct 13, 3:25 AM UTC	● No change
google/gemini-3.8-flash	31	775	135 h	1 · Oct 13, 1:56 AM UTC	● No change
google/gemini-3.8-flash (fp-v1, unknown backend)	1	28	< 1 h	–	◔ Collecting
google/gemini-3.8-flash (fp-v1, openrouter)	18	501	38 h	0.87 · Oct 6, 11:49 PM UTC	● No change
gpt-4.1	433	12,152	335 h	0.32 · Nov 7, 2:24 AM UTC	● No change
gpt-4.1 (fp-v1, unknown backend)	2	56	< 1 h	–	◔ Collecting
gpt-5.4	434	12,145	335 h	0.00067 · Nov 7, 2:23 AM UTC	● Changed
gpt-5.4-mini	433	12,139	332 h	0.019 · Nov 7, 2:24 AM UTC	● No change
gpt-5.4-mini (fp-v1, unknown backend)	2	56	< 1 h	–	◔ Collecting
gpt-5.4 (fp-v1, unknown backend)	2	56	< 1 h	–	◔ Collecting
gpt-6-astra	433	24,171	335 h	0.00067 · Nov 7, 2:22 AM UTC	● Changed
gpt-6-astra (fp-v1, unknown backend)	2	112	< 1 h	–	◔ Collecting
gpt-6-luna	437	24,392	335 h	0.00067 · Nov 7, 2:20 AM UTC	● Changed
gpt-6-luna (fp-v1, unknown backend)	2	112	< 1 h	–	◔ Collecting
gpt-6-sol	434	12,154	335 h	0.00067 · Nov 7, 2:23 AM UTC	● Changed
gpt-6-sol (fp-v1, unknown backend)	2	56	< 1 h	–	◔ Collecting
gpt-6.1-sol	435	24,229	332 h	0.00067 · Nov 7, 2:21 AM UTC	● Changed
gpt-6.1-sol (fp-v1, unknown backend)	2	112	< 1 h	–	◔ Collecting
ibm-granite/granite-4.2-8b	33	910	135 h	0.56 · Oct 13, 3:25 AM UTC	● No change
ibm-granite/granite-4.2-8b (fp-v1, openrouter, deepinfra/bf16)	18	504	36 h	0.5 · Oct 6, 11:49 PM UTC	● No change
inception/mercury-2.5	34	936	136 h	0.52 · Oct 13, 3:25 AM UTC	● No change
inception/mercury-2.5 (fp-v1, unknown backend)	1	28	< 1 h	–	◔ Collecting
inception/mercury-2.5 (fp-v1, openrouter)	18	441	38 h	0.79 · Oct 6, 11:49 PM UTC	● No change
meta-llama/llama-3.3-70b-instruct	52	1,419	130 h	0.9 · Oct 13, 3:25 AM UTC	● No change
meta-llama/llama-4-maverick	52	1,453	181 h	0.56 · Oct 13, 3:25 AM UTC	● No change
microsoft/phi-4	53	1,459	183 h	0.56 · Oct 13, 3:25 AM UTC	● No change
microsoft/phi-4 (fp-v1, unknown backend)	1	28	< 1 h	–	◔ Collecting
minimax/minimax-m3	31	789	132 h	0.97 · Oct 12, 11:25 PM UTC	● No change
minimax/minimax-m3 (fp-v1, openrouter, minimax/fp8)	18	504	36 h	0.94 · Oct 6, 11:49 PM UTC	● No change
mistralai/mistral-medium-3-5	50	1,280	127 h	0.84 · Oct 12, 11:25 PM UTC	● No change
mistralai/mistral-medium-3-5 (fp-v1, unknown backend)	1	28	< 1 h	–	◔ Collecting
mistralai/mistral-small-2603	52	1,121	132 h	0.039 · Oct 13, 3:25 AM UTC	● No change
mistralai/mistral-small-2603 (fp-v1, unknown backend)	1	28	< 1 h	–	◔ Collecting
moonshotai/kimi-k2.6	31	775	135 h	1 · Oct 13, 1:55 AM UTC	● No change
moonshotai/kimi-k2.6 (fp-v1, openrouter, moonshotai/int4)	18	503	36 h	0.75 · Oct 6, 11:49 PM UTC	● No change
nvidia/nemotron-3.5-lightning	34	935	136 h	1 · Oct 13, 3:25 AM UTC	● No change
nvidia/nemotron-3.5-lightning (fp-v1, openrouter, deepinfra/bf16)	19	507	36 h	0.79 · Oct 6, 11:49 PM UTC	● No change
qwen/qwen3.8-flash	51	1,421	182 h	0.84 · Oct 13, 3:25 AM UTC	● No change
qwen/qwen3.8-flash (fp-v1, unknown backend)	1	28	< 1 h	–	◔ Collecting
qwen/qwen3.8-max-0902	51	1,291	130 h	0.75 · Oct 13, 1:55 AM UTC	● No change
qwen/qwen3.8-max-0902 (fp-v1, unknown backend)	1	28	< 1 h	–	◔ Collecting
stepfun/step-3.7-flash	33	758	136 h	1 · Oct 13, 3:25 AM UTC	● No change
stepfun/step-3.7-flash (fp-v1, openrouter, stepfun/fp8)	18	504	36 h	0.75 · Oct 6, 11:49 PM UTC	● No change
tencent/hy3	34	819	136 h	0.46 · Oct 13, 3:25 AM UTC	● No change
tencent/hy3 (fp-v1, openrouter, tencent/fp8)	18	504	36 h	0.31 · Oct 6, 11:49 PM UTC	● No change
thinkingmachines/inkling	31	768	132 h	0.083 · Oct 12, 11:25 PM UTC	● No change
thinkingmachines/inkling (fp-v1, openrouter, together)	18	491	36 h	0.81 · Oct 6, 11:49 PM UTC	● No change
upstage/solar-pro4	52	1,452	183 h	0.58 · Oct 13, 3:25 AM UTC	● No change
upstage/solar-pro4 (fp-v1, unknown backend)	1	28	< 1 h	–	◔ Collecting
x-ai/grok-4.7	50	1,290	130 h	0.49 · Oct 13, 1:56 AM UTC	● No change
x-ai/grok-4.7 (fp-v1, unknown backend)	1	28	< 1 h	–	◔ Collecting
xiaomi/mimo-v2.6-pro	52	1,432	181 h	0.56 · Oct 13, 3:25 AM UTC	● No change
z-ai/glm-5.3	29	765	132 h	0.76 · Oct 12, 11:25 PM UTC	● No change
z-ai/glm-5.3 (fp-v1, openrouter, z-ai/fp8)	18	501	36 h	0.88 · Oct 6, 11:49 PM UTC	● No change

A change is flagged when adj. p stays below 0.01 for 2 checks in a row. Until an endpoint has 24 hours of history a flag is shown as provisional, not as an incident.

RECORD · THREE KINDS OF INCIDENT, EACH WITH ITS OWN RULE

Incidents

Three kinds of incident, each with its own rule. Scored tests: a Fisher exact test on pass counts against the endpoint’s baseline, corrected across all model × capability series (Benjamini–Hochberg), with a minimum change in points (either direction) and persistence across runs. Behaviour fingerprints: the exact permutation test above, corrected across endpoints, significant in two consecutive checks, and only after 24 hours of baseline. Reported identity: deterministic; opened as soon as the API reports a different model, with no p-value. Input-token counts: also deterministic, but opened only after a stable baseline and a change on at least a quarter of the fixed prompts in two consecutive runs. Each incident carries its change point and a permalink. Subscribe: Atom · JSON Feed.

● Opengpt-6-astraConfidence: Medium

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (1344 recent vs 6300 baseline responses). Biggest shifts: answers on “Name a famous scientist. Reply with only the name.”; answers on “Give one tip for sleeping better, in one sentence.”; wording on “Give one tip for sleeping better, in one sentence.”

Change began ≈ Nov 4, 9:14 PM UTC · adjusted p 6.7e-4 · still open · Permalink

● Opengpt-6-lunaConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (1344 recent vs 5906 baseline responses). Biggest shifts: whether it reasoned at all on “Give one tip for sleeping better, in one sentence.”; output length on “Give one tip for sleeping better, in one sentence.”; reasoning tokens on “Give one tip for sleeping better, in one sentence.”

Change began ≈ Nov 4, 8:54 AM UTC · adjusted p 6.7e-4 · still open · Permalink

● Opengpt-5.4Confidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 5082 baseline responses). Biggest shifts: token probabilities on “Name a famous scientist. Reply with only the name.”; token probabilities on “Say hello in a creative way, in one short sentence.”; token probabilities on “all prompts together”

Change began ≈ Oct 29, 1:57 PM UTC · adjusted p 6.7e-4 · still open · Permalink

✓ Resolvedgpt-6-astraConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (1316 recent vs 10136 baseline responses). Biggest shifts: visible output length on “Suggest a name for a pet cat. Reply with only the name.”; whether it reasoned at all on “Suggest a name for a pet cat. Reply with only the name.”; output length on “Suggest a name for a pet cat. Reply with only the name.”

Change began ≈ Oct 29, 1:56 PM UTC · adjusted p 4.0e-3 · resolved Nov 6 · Permalink

✓ Resolvedgpt-6-lunaConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (168 recent vs 10302 baseline responses). Biggest shifts: whether it reasoned at all on “Write the first sentence of a mystery novel.”; whether it reasoned at all on “all prompts together”; visible output length on “Name one musical instrument. Reply with one word.”

Change began ≈ Oct 28, 6:27 PM UTC · adjusted p 1.7e-3 · resolved Oct 29 · Permalink

● Opengpt-6-solConfidence: Medium

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 5292 baseline responses). Biggest shifts: reasoning tokens on “Complete this sentence: The old house at the end of the street”; whether it reasoned at all on “Complete this sentence: The old house at the end of the street”; output length on “Flip a coin. Reply with only heads or tails.”

Change began ≈ Oct 27, 2:44 AM UTC · adjusted p 6.7e-4 · still open · Permalink

● Openchat-latestConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 5236 baseline responses). Biggest shifts: wording on “Write the first sentence of a mystery novel.”; output length on “Write the first sentence of a mystery novel.”; answers on “List three uses for a brick.”

Change began ≈ Oct 26, 9:10 PM UTC · adjusted p 6.7e-4 · still open · Permalink

✓ Resolvedgpt-6-lunaConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (1342 recent vs 11760 baseline responses). Biggest shifts: whether it reasoned at all on “Give one tip for sleeping better, in one sentence.”; visible output length on “Give one tip for sleeping better, in one sentence.”; whether it reasoned at all on “Say hello in a creative way, in one short sentence.”

Change began ≈ Oct 23, 7:13 PM UTC · adjusted p 9.7e-3 · resolved Oct 26 · Permalink

✓ Resolvedchat-latestConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 6048 baseline responses). Biggest shifts: answers on “List three uses for a brick.”; output length on “List three uses for a brick.”; wording on “List three uses for a brick.”

Change began ≈ Oct 23, 3:09 AM UTC · adjusted p 6.5e-3 · resolved Oct 25 · Permalink

✓ Resolvedgpt-6-solConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 6076 baseline responses). Biggest shifts: reasoning tokens on “Write a haiku about autumn.”; output length on “Write a haiku about autumn.”; whether it reasoned at all on “Write a haiku about autumn.”

Change began ≈ Oct 23, 1:00 AM UTC · adjusted p 3.3e-3 · resolved Oct 26 · Permalink

✓ Resolvedgpt-5.4Confidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 5950 baseline responses). Biggest shifts: token probabilities on “Name one animal. Reply with one word.”; first token on “Write a haiku about autumn.”; token probabilities on “Give me a one-line motivational quote.”

Change began ≈ Oct 21, 7:10 PM UTC · adjusted p 5.3e-3 · resolved Nov 4 · Permalink

✓ Resolvedgpt-4.1Confidence: Medium

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (84 recent vs 5964 baseline responses). Biggest shifts: token probabilities on “Pick a day of the week. Reply with one word.”; first token on “Pick a day of the week. Reply with one word.”; token probabilities on “Suggest a name for a pet cat. Reply with only the name.”

Change began ≈ Oct 22, 9:39 PM UTC · adjusted p 6.7e-4 · resolved Nov 5 · Permalink

✓ Resolvedgpt-4.1Confidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 6580 baseline responses). Biggest shifts: token probabilities on “Describe a cat in exactly five words.”; token probabilities on “Give me a one-line motivational quote.”; token probabilities on “Pick a day of the week. Reply with one word.”

Change began ≈ Oct 20, 12:30 AM UTC · adjusted p 8.0e-3 · resolved Oct 21 · Permalink

✓ Resolvedgpt-6-solConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 6582 baseline responses). Biggest shifts: whether it reasoned at all on “Write a haiku about autumn.”; output length on “Write a haiku about autumn.”; whether it reasoned at all on “all prompts together”

Change began ≈ Oct 19, 7:59 PM UTC · adjusted p 6.4e-3 · resolved Oct 21 · Permalink

✓ Resolvedgpt-6-lunaConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (1344 recent vs 13278 baseline responses). Biggest shifts: whether it reasoned at all on “Write one sentence about the sea.”; visible output length on “Name one programming language. Reply with only its name.”; whether it reasoned at all on “Name one programming language. Reply with only its name.”

Change began ≈ Oct 19, 7:56 PM UTC · adjusted p 8.0e-4 · resolved Oct 21 · Permalink

✓ Resolvedgpt-6-solConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 6414 baseline responses). Biggest shifts: output length on “Give me a one-line motivational quote.”; visible output length on “Give me a one-line motivational quote.”; wording on “Write one sentence about rain.”

Change began ≈ Oct 18, 2:11 PM UTC · adjusted p 2.0e-3 · resolved Oct 20 · Permalink

✓ Resolvedgpt-6-lunaConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (1344 recent vs 12830 baseline responses). Biggest shifts: output length on “Name a famous scientist. Reply with only the name.”; output length on “Flip a coin. Reply with only heads or tails.”; reasoning tokens on “Flip a coin. Reply with only heads or tails.”

Change began ≈ Oct 18, 1:00 PM UTC · adjusted p 6.4e-3 · resolved Oct 20 · Permalink

✓ Resolvedgpt-6-lunaConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (1344 recent vs 11878 baseline responses). Biggest shifts: output length on “Name a famous scientist. Reply with only the name.”; reasoning tokens on “Name a famous scientist. Reply with only the name.”; visible output length on “Write one sentence about rain.”

Change began ≈ Oct 17, 6:21 PM UTC · adjusted p 9.0e-3 · resolved Oct 19 · Permalink

✓ Resolvedgpt-6-lunaConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (1344 recent vs 11206 baseline responses). Biggest shifts: output length on “Name a famous scientist. Reply with only the name.”; reasoning tokens on “Name a famous scientist. Reply with only the name.”; output length on “Pick a number between 1 and 1000. Reply with only the number.”

Change began ≈ Oct 17, 1:07 AM UTC · adjusted p 7.2e-3 · resolved Oct 18 · Permalink

✓ Resolvedgpt-6-solConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 4790 baseline responses). Biggest shifts: whether it reasoned at all on “Name a chemical element. Reply with one word.”; visible output length on “Name a chemical element. Reply with one word.”; reasoning tokens on “Name a chemical element. Reply with one word.”

Change began ≈ Oct 14, 11:11 AM UTC · adjusted p 9.3e-3 · resolved Oct 17 · Permalink

✓ Resolvedgpt-6-astraConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (168 recent vs 9975 baseline responses). Biggest shifts: visible output length on “all prompts together”; whether it reasoned at all on “all prompts together”; output length on “all prompts together”

Change began ≈ Oct 14, 11:07 PM UTC · adjusted p 8.0e-3 · resolved Oct 27 · Permalink

✓ Resolvedgpt-6-lunaConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (1344 recent vs 8742 baseline responses). Biggest shifts: whether it reasoned at all on “all prompts together”; visible output length on “Name one programming language. Reply with only its name.”; whether it reasoned at all on “Name one programming language. Reply with only its name.”

Change began ≈ Oct 13, 11:12 AM UTC · adjusted p 8.7e-3 · resolved Oct 15 · Permalink

✓ Resolvedgpt-6-astraConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (1344 recent vs 8071 baseline responses). Biggest shifts: output length on “Name one fruit. Reply with one word.”; reasoning tokens on “Name one fruit. Reply with one word.”; visible output length on “Name one fruit. Reply with one word.”

Change began ≈ Oct 12, 7:10 PM UTC · adjusted p 7.3e-3 · resolved Oct 15 · Permalink

● Opengpt-6.1-solConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (1344 recent vs 7961 baseline responses). Biggest shifts: output length on “Pick a day of the week. Reply with one word.”; reasoning tokens on “Pick a day of the week. Reply with one word.”; whether it reasoned at all on “all prompts together”

Change began ≈ Oct 12, 4:40 PM UTC · adjusted p 6.7e-4 · still open · Permalink

✓ Resolvedgpt-6-solConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 3614 baseline responses). Biggest shifts: reasoning tokens on “Write one sentence about rain.”; visible output length on “Write a haiku about autumn.”; whether it reasoned at all on “Describe a cat in exactly five words.”

Change began ≈ Oct 13, 4:35 AM UTC · adjusted p 2.4e-3 · resolved Oct 15 · Permalink

✓ Resolvedgpt-4.1Confidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 3584 baseline responses). Biggest shifts: token probabilities on “Give me one random English noun. Reply with one word.”; output length on “What is 17 * 23?”; answers on “Name one fruit. Reply with one word.”

Change began ≈ Oct 11, 9:29 PM UTC · adjusted p 5.2e-3 · resolved Oct 13 · Permalink

✓ Resolvedgpt-6-lunaConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (1344 recent vs 6558 baseline responses). Biggest shifts: whether it reasoned at all on “Name one fruit. Reply with one word.”; visible output length on “Name one color. Reply with one word.”; whether it reasoned at all on “Name one color. Reply with one word.”

Change began ≈ Oct 11, 6:14 AM UTC · adjusted p 4.8e-3 · resolved Oct 13 · Permalink

✓ Resolvedgpt-4.1Confidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 2744 baseline responses). Biggest shifts: first token on “Pick a random integer between 1 and 50. Reply with only the number.”; token probabilities on “Pick a random integer between 1 and 50. Reply with only the number.”; token probabilities on “Flip a coin. Reply with only heads or tails.”

Change began ≈ Oct 10, 1:42 AM UTC · adjusted p 3.3e-3 · resolved Oct 11 · Permalink

✓ Resolvedgpt-6-lunaConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (1344 recent vs 5270 baseline responses). Biggest shifts: whether it reasoned at all on “Explain what a variable is in programming, in two sentences.”; whether it reasoned at all on “Name one fruit. Reply with one word.”; visible output length on “Name one fruit. Reply with one word.”

Change began ≈ Oct 9, 11:26 PM UTC · adjusted p 1.7e-3 · resolved Oct 11 · Permalink

✓ Resolvedgpt-6.1-solConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (168 recent vs 5385 baseline responses). Biggest shifts: visible output length on “Name one programming language. Reply with only its name.”; whether it reasoned at all on “Name one programming language. Reply with only its name.”; output length on “Name one programming language. Reply with only its name.”

Change began ≈ Oct 9, 2:28 AM UTC · adjusted p 1.6e-3 · resolved Oct 13 · Permalink

✓ Resolvedgpt-6-astraConfidence: Medium

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (168 recent vs 5327 baseline responses). Biggest shifts: whether it reasoned at all on “all prompts together”; whether it reasoned at all on “Name one fruit. Reply with one word.”; visible output length on “Name one fruit. Reply with one word.”

Change began ≈ Oct 10, 12:37 AM UTC · adjusted p 6.7e-3 · resolved Oct 13 · Permalink

✓ Resolvedgpt-6-solConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 2018 baseline responses). Biggest shifts: whether it reasoned at all on “Suggest a name for a pet cat. Reply with only the name.”; output length on “Pick a day of the week. Reply with one word.”; reasoning tokens on “Pick a day of the week. Reply with one word.”

Change began ≈ Oct 8, 6:05 PM UTC · adjusted p 7.3e-3 · resolved Oct 10 · Permalink

✓ Resolvedgpt-5.4-miniConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 1919 baseline responses). Biggest shifts: output length on “Pick a random letter of the alphabet. Reply with only the letter.”; first token on “Give one tip for sleeping better, in one sentence.”; token probabilities on “Name a city in Europe. Reply with only the city name.”

Change began ≈ Oct 8, 1:32 PM UTC · adjusted p 8.7e-3 · resolved Nov 5 · Permalink

✓ Resolvedgpt-6.1-solConfidence: Medium

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (1297 recent vs 2968 baseline responses). Biggest shifts: output length on “Name one animal. Reply with one word.”; reasoning tokens on “Name one animal. Reply with one word.”; visible output length on “Name one animal. Reply with one word.”

Change began ≈ Oct 7, 8:53 PM UTC · adjusted p 2.0e-3 · resolved Oct 10 · Permalink

✓ Resolvedgpt-6-lunaConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (1294 recent vs 2688 baseline responses). Biggest shifts: output length on “all prompts together”; reasoning tokens on “all prompts together”; visible output length on “Name one programming language. Reply with only its name.”

Change began ≈ Oct 7, 12:38 PM UTC · adjusted p 3.2e-3 · resolved Oct 9 · Permalink

✓ Resolvedgpt-4.1Confidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 1260 baseline responses). Biggest shifts: token probabilities on “Say hello in a creative way, in one short sentence.”; answers on “Complete this sentence: The old house at the end of the street”; token probabilities on “Give one tip for sleeping better, in one sentence.”

Change began ≈ Oct 7, 7:37 AM UTC · adjusted p 6.0e-3 · resolved Oct 9 · Permalink

✓ Resolvedgpt-6-astraConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (1295 recent vs 2464 baseline responses). Biggest shifts: visible output length on “Give one tip for sleeping better, in one sentence.”; output length on “Give one tip for sleeping better, in one sentence.”; whether it reasoned at all on “Give one tip for sleeping better, in one sentence.”

Change began ≈ Oct 7, 7:35 AM UTC · adjusted p 4.0e-3 · resolved Oct 9 · Permalink

✓ Resolvedgpt-6-solConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 952 baseline responses). Biggest shifts: whether it reasoned at all on “all prompts together”; whether it reasoned at all on “Complete this sentence: The old house at the end of the street”; reasoning tokens on “Complete this sentence: The old house at the end of the street”

Change began ≈ Oct 6, 7:15 PM UTC · adjusted p 1.0e-2 · resolved Oct 8 · Permalink

✓ Resolvedchat-latestConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (84 recent vs 1484 baseline responses). Biggest shifts: answers on “Name one fruit. Reply with one word.”; output length on “Name one fruit. Reply with one word.”; wording on “Explain what a variable is in programming, in two sentences.”

Change began ≈ Oct 7, 7:45 PM UTC · adjusted p 8.4e-3 · resolved Oct 17 · Permalink

✓ Resolvedgpt-6.1-solConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (1344 recent vs 1008 baseline responses). Biggest shifts: whether it reasoned at all on “Name one animal. Reply with one word.”; visible output length on “Name one animal. Reply with one word.”; reasoning tokens on “Suggest a name for a pet cat. Reply with only the name.”

Change began ≈ Oct 5, 11:31 PM UTC · adjusted p 9.3e-3 · resolved Oct 7 · Permalink

✓ Resolvedgpt-6-solConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 476 baseline responses). Biggest shifts: whether it reasoned at all on “all prompts together”; visible output length on “Write a haiku about autumn.”; visible output length on “Give me one random English noun. Reply with one word.”

Change began ≈ Oct 5, 9:25 PM UTC · adjusted p 7.5e-3 · resolved Oct 7 · Permalink

✓ Resolvedgpt-5.4-miniConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 448 baseline responses). Biggest shifts: token probabilities on “Name a chemical element. Reply with one word.”; first token on “Say hello in a creative way, in one short sentence.”; token probabilities on “Pick a random letter of the alphabet. Reply with only the letter.”

Change began ≈ Oct 5, 8:22 PM UTC · adjusted p 7.2e-3 · resolved Oct 9 · Permalink

✓ Resolvedgpt-5.4Confidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 448 baseline responses). Biggest shifts: first token on “List three uses for a brick.”; token probabilities on “Pick a number between 1 and 1000. Reply with only the number.”; token probabilities on “Suggest a name for a pet cat. Reply with only the name.”

Change began ≈ Oct 5, 8:22 PM UTC · adjusted p 2.4e-3 · resolved Oct 24 · Permalink

METHOD IN BRIEF · FULL ACCOUNT ON THE METHODOLOGY PAGE

How the measurements are made

01 · PROBES

Fixed tests, fresh variables

Short templated tests with new random variables every run, so an endpoint cannot memorise them, plus a fixed set of open prompts whose answer distribution is the fingerprint.

02 · SCORING

Deterministic, in code

JSON schemas, exact match, arithmetic, planted needles, tool-call shape. A check passes or fails by rule; no model grades another model.

03 · INFERENCE

Each endpoint against itself

Scored tests: Fisher exact tests with Benjamini–Hochberg correction across series and a persistence rule. Fingerprints: exact permutation tests over whole runs, two consecutive significant checks, a 24-hour baseline. Reported identity: deterministic, opened at once. Input-token counts: deterministic, after a stable baseline and two consecutive runs.

Canary monitoring for your own endpoints.

Martingale runs scheduled deterministic tests against the endpoints you depend on, learns each one’s own baseline, and is built to tell you when it changes. It is a separate product from this public tracker, under development: alert delivery to Slack and email is in limited testing and no customer is relying on it yet.

Monitor your endpoints