Research · October 5, 2026

Same model, different host

A router sells one model id and sends each request to one of several hosts. We pinned each host in turn and sent the same prompts. On some models, which host answers decides how much the model reasons, how many tokens you are billed, and whether a hard task is solved. None of this shows up as a model change, because the model name never changes.

Every number on this page comes from one morning of measurements. Each table has its limits written next to it; please read them before quoting a host by name.

One Llama model id, four hosts

meta-llama/llama-3.3-70b-instruct on OpenRouter is served by 11 hosts. We pinned four: Novita (bf16), Parasail (fp8), Groq (undeclared) and DeepInfra turbo (fp8).

Each host pinned explicitly (provider.only, no fallbacks). The 28 Raven Alpha fingerprint prompts (fp-v1), 10 passes per host, default sampling, no system message, hosts interleaved pass by pass so time of day cannot explain a difference between hosts. Raven Alpha's exact permutation test over whole passes between every pair, words only; logprobs excluded because hosts report them differently. Benjamini-Hochberg adjusted p over the six pairs.

first run: 2026-10-05 04:10-05:18 UTC, 280 (270 from Groq) responses per host, raw data lost (see caveats) · replication: 2026-10-05 06:23-07:40 UTC, 280 responses per host, raw data kept and backed up. Threshold: adjusted p < 0.01.

Same-host control

Same-host control: each host tested against itself on random halves of its own passes, 40 splits per host. The count is how many of the 40 splits came out below the 0.01 threshold; a test that invents differences would fail this.

Why DeepInfra turbo differs

The same prompt is billed as 43 input tokens on Novita, Parasail and Groq and 18 on DeepInfra turbo, on every one of the 28 prompts. The Llama 3 chat template adds a default header with the knowledge cutoff and today's date when there is no system message; three hosts apply it and DeepInfra's turbo endpoint does not. Adding the header by hand makes DeepInfra answer exactly like the others, so this is a prompt-template difference, not different weights.

2026-10-05 05:20 UTC, 4 calls each.

Every host of eleven open models, asked once

86 hosts of 11 open models on OpenRouter were pinned in turn and asked the same 28 prompts once; 81 answered. 2026-10-05 06:36-07:45 UTC.

For each model, every host listed on the router (up to 16) was pinned in turn and asked the 28 Raven Alpha fingerprint prompts once. Output and reasoning tokens are as the host reports them through the router's usage field. Where hosts of one model spent very different amounts of output, pairs of hosts were given 30 identical hard problems (code tracing, seating puzzles, ledger replay) with an 8,000-12,000 token limit so a reasoning host is not cut short; answers are computed, not judged.

Models: Llama 4 Maverick, GLM 5.3, Inkling, Kimi K2.6, Hunyuan hy3, DeepSeek V4 Pro, Step 3.7 Flash, MiMo v2.6 Pro, Nemotron 3.5 Lightning, Granite 4.2 8B, MiniMax M3.

On four of eleven models, some hosts spend far less reasoning on easy prompts

No such split: Kimi K2.6 (14 hosts), Hunyuan hy3, MiMo v2.6 Pro, Step 3.7 Flash, Granite 4.2 8B, Inkling, Llama 4 Maverick.

It decides whether hard problems get solved

"Strict" is the bench's scoring: the reply must be only the answer, as the prompt asks. "Lenient" counts a reply that contains the right answer anywhere, working shown or not.

Hosts bill different input-token counts for the same request

Input tokens for the same prompts are deterministic. 5 of 11 models have hosts that disagree.

Availability

5 of 86 pinned hosts did not answer (5xx, 429, or removed by the account's data policy), among them CoreWeave for Kimi K2.6 and MiniMax M3, which the router listed with over 99.9% uptime at the time.

What this does and does not show

The other two models in the four-host comparison

• MiniMax M3 (four hosts, 7-8 passes each). In words, no pair of hosts differs (adjusted p 0.41-0.92), including fp4 against fp8. Token accounting does differ: for the same one-word answer, MiniMax's own host reports 1 visible output token and Together reports 4, and DeepInfra and CoreWeave sometimes report more reasoning tokens than total output tokens. A billing and metering difference, not a behaviour difference. All four bill the same 184 input tokens for "What is 17 * 23?".
• DeepSeek V4.1 Flash. Not enough data. CoreWeave and Decart returned 5xx errors for long stretches when pinned and the collection was stopped at 4-5 passes per host, too few for the test. DeepSeek's own host could not be used: the router account's data policy excludes it.

What we changed because of this

Each multi-host model on this bench is now pinned to one host, so a change in the router's host mix cannot show up as model drift. How the bench pins hosts is described in the methodology.

Sources: docs/research/2026-10-05-same-model-different-host.md; docs/research/2026-10-05-host-census.md at commit 844b57a. Data: hosts.json.