{
 "generatedAt": "2026-10-05T08:13:00.000Z",
 "sources": [
  {
   "label": "Same model name, four hosts: one behaves differently",
   "path": "docs/research/2026-10-05-same-model-different-host.md",
   "task": "T13"
  },
  {
   "label": "Host census: one model name, many configurations",
   "path": "docs/research/2026-10-05-host-census.md",
   "task": "T13"
  }
 ],
 "hosts": [
  "Novita",
  "Parasail",
  "Groq",
  "DeepInfra",
  "DeepInfra turbo",
  "DigitalOcean",
  "Reka",
  "Azure",
  "Wafer",
  "Relace",
  "GMICloud",
  "CoreWeave",
  "MiniMax",
  "Cloudflare",
  "SambaNova",
  "Venice",
  "StreamLake",
  "Together",
  "Decart",
  "Z.AI"
 ],
 "pairwise": {
  "title": "One Llama model id, four hosts",
  "modelId": "meta-llama/llama-3.3-70b-instruct",
  "router": "OpenRouter",
  "hostsListed": 11,
  "hosts": [
   {
    "name": "Novita",
    "declared": "bf16"
   },
   {
    "name": "Parasail",
    "declared": "fp8"
   },
   {
    "name": "Groq",
    "declared": "undeclared"
   },
   {
    "name": "DeepInfra turbo",
    "declared": "fp8"
   }
  ],
  "method": "Each host pinned explicitly (provider.only, no fallbacks). The 28 Raven Alpha fingerprint prompts (fp-v1), 10 passes per host, default sampling, no system message, hosts interleaved pass by pass so time of day cannot explain a difference between hosts. Raven Alpha's exact permutation test over whole passes between every pair, words only; logprobs excluded because hosts report them differently. Benjamini-Hochberg adjusted p over the six pairs.",
  "runs": [
   {
    "label": "first run",
    "window": "2026-10-05 04:10-05:18 UTC",
    "responsesPerHost": "280 (270 from Groq)",
    "rawData": "lost (see caveats)"
   },
   {
    "label": "replication",
    "window": "2026-10-05 06:23-07:40 UTC",
    "responsesPerHost": "280",
    "rawData": "kept and backed up"
   }
  ],
  "threshold": 0.01,
  "pairs": [
   {
    "a": "DeepInfra turbo",
    "b": "Parasail",
    "pFirst": 0.0003,
    "pReplication": 0.0003,
    "whatDiffers": "\"What is 17 * 23?\": step-by-step working vs \"17 * 23 = 391.\""
   },
   {
    "a": "DeepInfra turbo",
    "b": "Groq",
    "pFirst": 0.0004,
    "pReplication": 0.0004,
    "whatDiffers": "same"
   },
   {
    "a": "DeepInfra turbo",
    "b": "Novita",
    "pFirst": 0.0006,
    "pReplication": 0.0006,
    "whatDiffers": "same"
   },
   {
    "a": "Parasail",
    "b": "Groq",
    "pFirst": 0.0065,
    "pReplication": 0.14,
    "whatDiffers": "small differences spread over several prompts; did not replicate"
   },
   {
    "a": "Novita",
    "b": "Parasail",
    "pFirst": 0.007,
    "pReplication": 0.073,
    "whatDiffers": "\"List three uses for a brick\": Novita bolds the list items 90% of the time, Parasail 30%; did not replicate"
   },
   {
    "a": "Novita",
    "b": "Groq",
    "pFirst": 0.032,
    "pReplication": 0.091,
    "whatDiffers": "not significant"
   }
  ],
  "controls": {
   "description": "Same-host control: each host tested against itself on random halves of its own passes, 40 splits per host. The count is how many of the 40 splits came out below the 0.01 threshold; a test that invents differences would fail this.",
   "splits": 40,
   "first": [
    {
     "host": "Novita",
     "below": 0
    },
    {
     "host": "Parasail",
     "below": 0
    },
    {
     "host": "Groq",
     "below": 0
    },
    {
     "host": "DeepInfra turbo",
     "below": 1
    }
   ],
   "replication": [
    {
     "host": "Novita",
     "below": 0
    },
    {
     "host": "Parasail",
     "below": 0
    },
    {
     "host": "Groq",
     "below": 0
    },
    {
     "host": "DeepInfra turbo",
     "below": 0
    }
   ]
  },
  "explanation": {
   "summary": "The same prompt is billed as 43 input tokens on Novita, Parasail and Groq and 18 on DeepInfra turbo, on every one of the 28 prompts. The Llama 3 chat template adds a default header with the knowledge cutoff and today's date when there is no system message; three hosts apply it and DeepInfra's turbo endpoint does not. Adding the header by hand makes DeepInfra answer exactly like the others, so this is a prompt-template difference, not different weights.",
   "window": "2026-10-05 05:20 UTC, 4 calls each",
   "rows": [
    {
     "host": "DeepInfra turbo",
     "request": "user message only",
     "inputTokens": 18,
     "answer": "\"## Step 1: Multiply the numbers 17 and 23 ...\" (60-109 tokens), 4 of 4"
    },
    {
     "host": "DeepInfra turbo",
     "request": "plus system message \"Cutting Knowledge Date: December 2023 / Today Date: 26 Jul 2024\"",
     "inputTokens": 42,
     "answer": "\"17 * 23 = 391.\" 4 of 4"
    },
    {
     "host": "Parasail",
     "request": "user message only",
     "inputTokens": 43,
     "answer": "\"17 * 23 = 391.\" 4 of 4"
    }
   ]
  },
  "caveats": [
   "One session, four of the eleven Llama hosts, 10 passes each. The DeepInfra turbo difference replicated exactly; the two smaller differences (Parasail vs Groq, Novita vs Parasail) did not, so treat Novita, Parasail and Groq as indistinguishable in what they write until shown otherwise.",
   "This does not show that any host serves the wrong model, and it does not show a quantization effect. Declared precision differs (bf16, fp8, undeclared) and so may sampling defaults; the experiment cannot separate those causes.",
   "Words only. With token counts included all six pairs differ, but token counts can differ between hosts for accounting reasons alone, so the words-only numbers are the ones to quote.",
   "The raw rows of the first run were deleted by mistake on 2026-10-05 and cannot be re-derived; the replication's rows are kept. The first-run numbers were computed before the loss.",
   "One of Novita's 280 responses degenerated into repeated exclamation marks. One observation."
  ]
 },
 "census": {
  "title": "Every host of eleven open models, asked once",
  "router": "OpenRouter",
  "window": "2026-10-05 06:36-07:45 UTC",
  "modelsCount": 11,
  "hostsAsked": 86,
  "hostsAnswered": 81,
  "promptsPerHost": 28,
  "models": [
   "Llama 4 Maverick",
   "GLM 5.3",
   "Inkling",
   "Kimi K2.6",
   "Hunyuan hy3",
   "DeepSeek V4 Pro",
   "Step 3.7 Flash",
   "MiMo v2.6 Pro",
   "Nemotron 3.5 Lightning",
   "Granite 4.2 8B",
   "MiniMax M3"
  ],
  "method": "For each model, every host listed on the router (up to 16) was pinned in turn and asked the 28 Raven Alpha fingerprint prompts once. Output and reasoning tokens are as the host reports them through the router's usage field. Where hosts of one model spent very different amounts of output, pairs of hosts were given 30 identical hard problems (code tracing, seating puzzles, ledger replay) with an 8,000-12,000 token limit so a reasoning host is not cut short; answers are computed, not judged.",
  "reasoningSplit": {
   "lede": "On four of eleven models, some hosts spend far less reasoning on easy prompts",
   "rows": [
    {
     "model": "DeepSeek V4 Pro",
     "answered": 16,
     "lowHosts": [
      "DeepInfra",
      "DigitalOcean",
      "Reka",
      "Azure"
     ],
     "lowOutput": "455-605 tokens, 0 reasoning",
     "restOutput": "1,694-2,891 tokens, 1,160-2,332 reasoning"
    },
    {
     "model": "GLM 5.3",
     "answered": 15,
     "lowHosts": [
      "Wafer (two endpoints)",
      "Reka (two endpoints)",
      "Relace"
     ],
     "lowHostCount": 5,
     "lowOutput": "532-743 tokens, 16-215 reasoning",
     "restOutput": "3,292-5,520 tokens, 3,112-5,300 reasoning"
    },
    {
     "model": "Nemotron 3.5 Lightning",
     "answered": 5,
     "lowHosts": [
      "DeepInfra"
     ],
     "lowOutput": "564 tokens, 0 reasoning",
     "restOutput": "8,770-9,495 tokens, 8,569-9,051 reasoning"
    },
    {
     "model": "MiniMax M3",
     "answered": 12,
     "lowHosts": [
      "GMICloud"
     ],
     "lowOutput": "531 tokens, 0 reasoning",
     "restOutput": "1,017-2,400 tokens, 527-1,738 reasoning"
    }
   ],
   "noSplit": [
    "Kimi K2.6 (14 hosts)",
    "Hunyuan hy3",
    "MiMo v2.6 Pro",
    "Step 3.7 Flash",
    "Granite 4.2 8B",
    "Inkling",
    "Llama 4 Maverick"
   ],
   "caveats": [
    "\"Far less reasoning\" is inferred from output-token counts on the 28 easy census prompts. A host may consider it a default setting, and reasoning may be switchable per request on those hosts; request parameters were not tested.",
    "GLM 5.3 is different. Its five low-output hosts spent almost no reasoning on the 28 easy census prompts, but on the 30 hard problems they reason and score like the maker's own host (Z.AI 20 / 30, Reka 22 / 30, Wafer 22 / 30; see the hard-task table). So for GLM the hosts differ in how much they reason on easy prompts, not in whether they can solve hard ones.",
    "One pass of 28 prompts per host, one morning. Token counts are as reported through the router's usage field."
   ]
  },
  "hardTasks": {
   "lede": "It decides whether hard problems get solved",
   "scoring": "\"Strict\" is the bench's scoring: the reply must be only the answer, as the prompt asks. \"Lenient\" counts a reply that contains the right answer anywhere, working shown or not.",
   "rows": [
    {
     "model": "Nemotron 3.5 Lightning",
     "host": "CoreWeave",
     "strict": "17 / 30",
     "lenient": null,
     "outputTokens": 207184
    },
    {
     "model": "Nemotron 3.5 Lightning",
     "host": "DeepInfra",
     "strict": "0 / 30",
     "lenient": null,
     "outputTokens": 12847
    },
    {
     "model": "MiniMax M3",
     "host": "MiniMax (the maker's own)",
     "strict": "21 / 30",
     "lenient": null,
     "outputTokens": 113750
    },
    {
     "model": "MiniMax M3",
     "host": "GMICloud",
     "strict": "0 / 30",
     "lenient": null,
     "outputTokens": 15531
    },
    {
     "model": "DeepSeek V4 Pro",
     "host": "Parasail",
     "strict": "23 / 30",
     "lenient": null,
     "outputTokens": 110804
    },
    {
     "model": "DeepSeek V4 Pro",
     "host": "DigitalOcean",
     "strict": "0 / 30 (twice)",
     "lenient": "15 / 30",
     "outputTokens": 17808
    },
    {
     "model": "DeepSeek V4 Pro",
     "host": "Reka",
     "strict": "0 / 30",
     "lenient": "19 / 30",
     "outputTokens": 30411
    },
    {
     "model": "DeepSeek V4 Pro",
     "host": "DeepInfra",
     "strict": "0 / 30",
     "lenient": "17 / 30",
     "outputTokens": 22024
    },
    {
     "model": "DeepSeek V4 Pro",
     "host": "Azure",
     "strict": "0 / 30",
     "lenient": "16 / 30",
     "outputTokens": 19410
    },
    {
     "model": "GLM 5.3",
     "host": "Z.AI (the maker's own)",
     "strict": "20 / 30",
     "lenient": null,
     "outputTokens": 104587
    },
    {
     "model": "GLM 5.3",
     "host": "Reka (low output in the census)",
     "strict": "22 / 30",
     "lenient": null,
     "outputTokens": 103031
    },
    {
     "model": "GLM 5.3",
     "host": "Wafer (low output in the census)",
     "strict": "22 / 30",
     "lenient": null,
     "outputTokens": 76963
    }
   ],
   "caveats": [
    "Lenient scores were recorded for the second DeepSeek run only.",
    "GMICloud (MiniMax M3) and DeepInfra (Nemotron) replied with a bare answer in the right format, and it was wrong: those zeros are real failures. The four low-output DeepSeek hosts write their working into the visible reply instead; they never follow the \"only the answer\" instruction (0 of 30 each), and with lenient scoring they are right 50-63% of the time against 77% strict for the reasoning host.",
    "GLM 5.3 is different: its low-output census hosts reason and score like the maker's own host on the 30 hard problems (20-22 / 30). Low output in the census is a reason to look, not a verdict: of the four models flagged, three showed a real gap on hard problems and one did not.",
    "24-30 tasks per host. Answers are computed by running the program or solving the puzzle, never judged."
   ]
  },
  "inputTokens": {
   "lede": "Hosts bill different input-token counts for the same request",
   "intro": "Input tokens for the same prompts are deterministic. 5 of 11 models have hosts that disagree.",
   "rows": [
    {
     "model": "DeepSeek V4 Pro",
     "common": "412 tokens (13 hosts)",
     "outliers": "GMICloud +790 (about 28 per request), Novita +1,975 (about 71), Cloudflare +2,212 (about 79)"
    },
    {
     "model": "MiniMax M3",
     "common": "4,860 (8 hosts)",
     "outliers": "Novita -65, GMICloud -338, SambaNova -1,274, Venice -1,352"
    },
    {
     "model": "Kimi K2.6",
     "common": "525 (12 hosts)",
     "outliers": "StreamLake -11, Novita -28"
    },
    {
     "model": "Llama 4 Maverick",
     "common": "580 (2 hosts)",
     "outliers": "Novita +90"
    },
    {
     "model": "Inkling",
     "common": "692 (1 host)",
     "outliers": "Together +28"
    }
   ],
   "caveats": [
    "A different count means a different chat template, tokenizer or hidden system prompt, or different metering. Which one was not determined, except for Llama 3.3 on DeepInfra turbo above (a missing date header, confirmed by adding it back).",
    "Differences are totals over the 28 prompts, as billed through the router."
   ]
  },
  "availability": "5 of 86 pinned hosts did not answer (5xx, 429, or removed by the account's data policy), among them CoreWeave for Kimi K2.6 and MiniMax M3, which the router listed with over 99.9% uptime at the time.",
  "caveats": [
   "It shows that a router model id is not one configuration. Which host answers can decide how much the model reasons, how many tokens you are billed, and whether a hard task is solved. With default routing that choice is made per request.",
   "It does not show that any host serves different weights, and it does not show a quantization effect.",
   "Prior work: Gao et al. (2025), \"Model Equality Testing\", found 11 of 31 Llama endpoints differed from reference weights. This is a 2026 measurement with a different lens (configuration and metering), not a new method."
  ]
 },
 "otherModels": [
  {
   "model": "MiniMax M3 (four hosts, 7-8 passes each)",
   "text": "In words, no pair of hosts differs (adjusted p 0.41-0.92), including fp4 against fp8. Token accounting does differ: for the same one-word answer, MiniMax's own host reports 1 visible output token and Together reports 4, and DeepInfra and CoreWeave sometimes report more reasoning tokens than total output tokens. A billing and metering difference, not a behaviour difference. All four bill the same 184 input tokens for \"What is 17 * 23?\"."
  },
  {
   "model": "DeepSeek V4.1 Flash",
   "text": "Not enough data. CoreWeave and Decart returned 5xx errors for long stretches when pinned and the collection was stopped at 4-5 passes per host, too few for the test. DeepSeek's own host could not be used: the router account's data policy excludes it."
  }
 ],
 "benchChange": "Each multi-host model on this bench is now pinned to one host, so a change in the router's host mix cannot show up as model drift.",
 "sourceCommit": "844b57a"
}