MartingaleBench

← All endpoints

OpenAI · via openai · published window from November 3, 2026

gpt-6-sol: stability, speed and price

site endpoint id gpt-6-sol

Historical measurements. No successful scored check in the past 24 hours. Scores, latency and cost below describe the 24-hour window ending at the last successful check, not current performance. Current rankings exclude this endpoint.

Last successful scored check: November 3, 2026 9:04 PM UTC

Core pass rate—

100.0%

historical: latest measured day

Speed—

1.1s

historical median per canary call · p95 3.5s

Cost—

$0.123

historical observed mean per scored run · probe mix differs by endpoint

Behaviour fingerprint434 runs

Change detected

12,154 responses · last adj. p 0.00067 · Nov 7 2:23 AM UTC

Summary

gpt-6-sol has no successful scored check in the past 24 hours. The last measured day ended at 2026-11-03T21:04:49.753Z; 100.0% of core checks passed. These historical measurements are excluded from current rankings.

During that historical measurement window, median canary latency was 1.1s (p95 3.5s) and the observed mean cost per scored run was $0.123. This does not describe the endpoint’s present performance.

Raven Alpha has 12,154 fingerprint responses over 434 runs in this endpoint’s selected series (1 other partition of history exist and are not shown here); the last recorded state is “change detected” at 2026-11-07T02:23:45.103Z with an adjusted p of 0.00067.

9 incidents on record, 1 open.

Written from the numbers on this page; every figure is in results.json.

Measurement record

Provider: OpenAI
Route: openai
Model requested: gpt-6-sol
Model id returned: gpt-6-sol
Scored calls in window: 30
Fingerprint responses: 12,154
Core score change: —
Status: Historical · no recent check
Suite: hard-v2 + starter-v2 · methodology v0.2

Capabilities over time

Collecting dataTrend charts appear once there are two 6-hour periods of canary runs.

Historical measurement window by capability

Structured JSON100.0%

Schema-compliance regressions

Instruction following100.0%

Exact-format instruction drift

Arithmetic, answer only100.0%

Wrong arithmetic, or a model that starts showing its working when told to give only the number

Tool calling100.0%

Tool-use format regressions

Over-refusal100.0%

New safety layers refusing benign requests

Max-tokens respect100.0%

Truncation and limit changes

Incidents

● Opengpt-6-solConfidence: Medium

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 5292 baseline responses). Biggest shifts: reasoning tokens on “Complete this sentence: The old house at the end of the street”; whether it reasoned at all on “Complete this sentence: The old house at the end of the street”; output length on “Flip a coin. Reply with only heads or tails.”

Change began ≈ Oct 27, 2:44 AM UTC · adjusted p 6.7e-4 · still open · Permalink

✓ Resolvedgpt-6-solConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 6076 baseline responses). Biggest shifts: reasoning tokens on “Write a haiku about autumn.”; output length on “Write a haiku about autumn.”; whether it reasoned at all on “Write a haiku about autumn.”

Change began ≈ Oct 23, 1:00 AM UTC · adjusted p 3.3e-3 · resolved Oct 26 · Permalink

✓ Resolvedgpt-6-solConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 6582 baseline responses). Biggest shifts: whether it reasoned at all on “Write a haiku about autumn.”; output length on “Write a haiku about autumn.”; whether it reasoned at all on “all prompts together”

Change began ≈ Oct 19, 7:59 PM UTC · adjusted p 6.4e-3 · resolved Oct 21 · Permalink

✓ Resolvedgpt-6-solConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 6414 baseline responses). Biggest shifts: output length on “Give me a one-line motivational quote.”; visible output length on “Give me a one-line motivational quote.”; wording on “Write one sentence about rain.”

Change began ≈ Oct 18, 2:11 PM UTC · adjusted p 2.0e-3 · resolved Oct 20 · Permalink

✓ Resolvedgpt-6-solConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 4790 baseline responses). Biggest shifts: whether it reasoned at all on “Name a chemical element. Reply with one word.”; visible output length on “Name a chemical element. Reply with one word.”; reasoning tokens on “Name a chemical element. Reply with one word.”

Change began ≈ Oct 14, 11:11 AM UTC · adjusted p 9.3e-3 · resolved Oct 17 · Permalink

✓ Resolvedgpt-6-solConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 3614 baseline responses). Biggest shifts: reasoning tokens on “Write one sentence about rain.”; visible output length on “Write a haiku about autumn.”; whether it reasoned at all on “Describe a cat in exactly five words.”

Change began ≈ Oct 13, 4:35 AM UTC · adjusted p 2.4e-3 · resolved Oct 15 · Permalink

✓ Resolvedgpt-6-solConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 2018 baseline responses). Biggest shifts: whether it reasoned at all on “Suggest a name for a pet cat. Reply with only the name.”; output length on “Pick a day of the week. Reply with one word.”; reasoning tokens on “Pick a day of the week. Reply with one word.”

Change began ≈ Oct 8, 6:05 PM UTC · adjusted p 7.3e-3 · resolved Oct 10 · Permalink

✓ Resolvedgpt-6-solConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 952 baseline responses). Biggest shifts: whether it reasoned at all on “all prompts together”; whether it reasoned at all on “Complete this sentence: The old house at the end of the street”; reasoning tokens on “Complete this sentence: The old house at the end of the street”

Change began ≈ Oct 6, 7:15 PM UTC · adjusted p 1.0e-2 · resolved Oct 8 · Permalink

✓ Resolvedgpt-6-solConfidence: Low

Raven Alpha (behaviour fingerprint): Responses to fixed prompts no longer match this model’s own history (672 recent vs 476 baseline responses). Biggest shifts: whether it reasoned at all on “all prompts together”; visible output length on “Write a haiku about autumn.”; visible output length on “Give me one random English noun. Reply with one word.”

Change began ≈ Oct 5, 9:25 PM UTC · adjusted p 7.5e-3 · resolved Oct 7 · Permalink

Raven Alpha: what it answers

40 runs, 1,120 responses, Nov 4 2:34 PM UTC to Nov 7 2:23 AM UTC. Each row is one fixed prompt with no right answer; the detector tracks the distribution of answers, not any single one. Filled cells: that run's most common answer was the overall favourite. Hollow cells: it was something else (normal for prompts with many good answers).

Prompt	Most common answers (share of all responses)	Last 24 runs	Median tokens
Pick a random integer between 1 and 50. Reply with only the number. int50 · short answer	98%37 3%27		22
Pick a number between 1 and 1000. Reply with only the number. int1000 · short answer	33%527 15%537 10%731 13 distinct answers in 40 responses		4
Name one color. Reply with one word. color · short answer	100%Blue		4
Name one animal. Reply with one word. animal · short answer	85%Otter 8%Elephant 5%Tiger 4 distinct answers in 40 responses		5
Name a city in Europe. Reply with only the city name. city · short answer	55%Prague 40%Paris 5%Vienna		5
Give me one random English noun. Reply with one word. noun · short answer	98%Lantern 3%Lighthouse		22
Flip a coin. Reply with only heads or tails. coin · short answer	80%tails 20%heads		20
Pick a random letter of the alphabet. Reply with only the letter. letter · short answer	68%Q 13%M 10%R 5 distinct answers in 40 responses		18
Name one fruit. Reply with one word. fruit · short answer	100%Apple		4
Suggest a name for a pet cat. Reply with only the name. cat · short answer	93%Mochi 8%Miso		6
Name one programming language. Reply with only its name. lang · short answer	100%Python		4
Name a chemical element. Reply with one word. element · short answer	73%Oxygen 18%Neon 3%Hydrogen 6 distinct answers in 40 responses		16
Give one adjective that describes the ocean. Reply with one word. ocean · short answer	100%Vast		5
Name a famous scientist. Reply with only the name. scientist · short answer	90%Albert Einstein 10%Marie Curie		5
Pick a day of the week. Reply with one word. weekday · short answer	75%Tuesday 15%Thursday 8%Wednesday 4 distinct answers in 40 responses		4
Name one musical instrument. Reply with one word. instrument · short answer	95%Piano 3%Guitar 3%Violin		5
Write one sentence about rain. rain · free text	80%Rain tapped softly against the… 18%Rain taps softly against the… 3%Rain pattered softly against the…		10
Explain what a variable is in programming, in two sentences. variable · free text	95%A variable is a named… 5%A variable is a name…		38
List three uses for a brick. brick · free text	75%1. Building a wall 2… 8%1. Building walls or paths… 5%1. Build a wall. 2… 7 distinct answers in 40 responses		24
*What is 17 23?** mult · free text	100%391…		4
Write one sentence about the sea. sea · free text	63%The sea shimmered beneath the… 20%The sea stretches to the… 5%The sea glimmers beneath the… 7 distinct answers in 40 responses		12
Complete this sentence: The old house at the end of the street house · free text	100%The old house at the…		23
Describe a cat in exactly five words. catfive · free text	28%A curious, whiskered, purring companion… 8%A curious, whiskered little companion… 5%A curious, soft, whiskered companion… 24 distinct answers in 40 responses		46
Give me a one-line motivational quote. quote · free text	23%“Keep going—every small step moves… 18%“Start where you are, use… 13%“Progress begins the moment you… 16 distinct answers in 40 responses		14
Say hello in a creative way, in one short sentence. hello · free text	43%Hello, bright spark—may your day… 5%A little spark of sunshine… 5%A pocketful of sunshine says… 21 distinct answers in 40 responses		15
Write a haiku about autumn. haiku · free text	50%Amber leaves drift down Cool… 13%Amber leaves descend Cool winds… 5%Red leaves drift at dusk… 14 distinct answers in 40 responses		129
Write the first sentence of a mystery novel. story · free text	10%… 5%On the morning the lake… 5%My mother had been dead… 31 distinct answers in 40 responses		255
Give one tip for sleeping better, in one sentence. tip · free text	85%Keep a consistent bedtime and… 10%Go to bed and wake… 5%Keep a consistent sleep and…		16

Raw file: gpt-6-sol.json (the same reference martingale check compares your endpoint against).

Each category is scored by deterministic checks only (Structured JSON, Tool calling and others). Read the methodology.