Here is a serving bug I want you to sit with for a minute: a request’s answer depended on who arrived with it.
Not its latency. Not its position in the queue. Its answer. Under cross-request prefill batching, a MoE request’s own expert routing changed as a function of how many tokens its batch-mates brought along. Same prompt, same weights, same greedy decoding — different experts consulted, different tokens out, depending on the co-arrivals it happened to share a batch with.
This post is the anatomy of that defect: how a byte-compare gate caught it, how a differential hunt cornered it to two F32 GEMVs, what the fix guarantees now, and the one bounded exception that survives the fix — because a determinism contract you state without its exceptions is marketing, not a contract.
The gate that noticed
The serve gate is simple by design: replay the same 16 prompts at concurrency 1 and concurrency 16, greedy decoding, and byte-compare every output stream. Not “semantically similar.” Not “same score on a benchmark.” Byte-identical, per stream.
On 2026-08-02 that gate failed on two models. Ornith-35B matched on 6 of 16 streams; KAT-Coder matched on 7 of 16 (serve-gate-matrix.jsonl). Ten of sixteen requests to the Ornith model were getting different output when they arrived inside a batch than when they arrived alone.
The instinct with a failure like this is to blame nondeterminism — some atomics-order race, some scheduler flake. The differential razor said no. Batch runs were bit-identical run-to-run. Changing the content of the co-arriving prompt changed nothing. Changing KV offsets, positions, masks, rope — all exonerated, each with a bit-identical A/B (findings.jsonl, invariance_razor row). The only variable that mattered was the total number of tokens in the batched prefill call. This was deterministic — deterministically wrong, as a function of batch size.
m-scan: the staircase
So scan m. Hold one 19-token request fixed, grow the co-arrival token count, and compare the fixed request’s logits against its solo run at every step (mscan-o35b-p6.log):
- total m from 35 through 64: bit-identical to solo, maxdiff 0.0, every point.
- at m=65: logits jump, maxdiff 1.079. Same argmax, for now.
- at m=75: maxdiff 1.148 and the argmax flips — token 90700 becomes token 8160.
Piecewise-constant plateaus with sharp thresholds at m=65 and m=75, reproduced by a descending sweep (so not warm-up state, not process history). That staircase shape is a fingerprint: a GEMM library switching internal algorithms — split-k factor, tile shape, reduction order — as the problem shape crosses heuristic thresholds. Each algorithm is a different floating-point summation order, each summation order a slightly different answer.
The GEMM razor then checked every weight in the trunk for m-invariance, one at a time. Thirty-six weight matrices across four probed layers — attention, GDN, shared-expert up/down, lm_head — were bit-identical between m=74 and m=75. Two operations were not: the MoE prefill router (ffn_gate_inp, a 256-output F32 GEMV riding cuBLASLt) moved its rows by up to 3.9e-3 between m=19 and m=65, and the shared-expert gate dot (one output) moved 1.07e-4 between m=74 and m=75.
3.9e-3 is a small number. Feed it to top_k(8 of 256) and it stops being small: top-k is a discontinuity, and a hair’s difference in the 8th-vs-9th expert score selects a different expert, which contributes a completely different FFN output, which compounds through 40 layers.
16% of routing decisions
The route trace made the damage concrete. Compare the fixed request’s expert routing, solo versus batched at total m=75, across 760 (layer, token) routing decisions (trace-solo.txt / trace-batch.txt):
- 422 identical,
- 217 differing only in expert order (same set, different ranking — which still reweights the mixture),
- 121 — 16% — selecting a different expert set entirely.
First set difference: layer 3, token 6, expert 39 replaced by expert 157. Because some other request’s tokens were in the same prefill call.
One more finding is worth stating because it killed a comfortable theory. The hypothesis was that our post-trained onboards had unusually tight routing margins — that a well-trained base model would be safe. Refuted. The supported, stock Qwen3.6-35B control had the tightest prefill routing margins of the three models measured (min 0.069 vs Ornith’s 0.207), carried the exact same m=65/75 thresholds, and passed its serve gate 16/16 anyway — by prompt luck. Its near-ties simply didn’t land where the staircase stepped. The defect was in every MoE model on the engine; two models made it visible and one hid it. A gate that passes can still be sitting on the bug — which is an argument for gates that byte-compare rather than eyeball, and for controls.
The fix: m-invariant twins
The decode path never had this problem, because decode’s router and gate ride in-house kernels — router_gemv and sigmoid_dot_rows — with a fixed reduction order per row, independent of how many rows share the launch. The razor confirmed it: router_gemv, m=19 versus m=79, rows bit-identical, maxdiff 0.0.
So the fix (merge b3a5465f) is not a new kernel. It routes prefill’s router and all five shared-expert-gate arms through the m-invariant twins decode and spec-verify already use — one kernel class per op, for every token count, so dispatch choice can never change bits. Default on; MEMRA_ROUTER_PREFILL_EXACT=0 remains as a numeric rollback seam, documented as forfeiting the isolation guarantee (FLAGS.md).
Post-fix gates: Ornith-35B serve gate 6/16 → 16/16, KAT 7/16 → 16/16, the 9B and the Qwen control still 16/16; engine m-invariance 60/60 across m=35..94; kernel-check, decode-batch, prime-batch, spec K=1..8 all green. The prefill throughput cost of leaving cuBLASLt was mostly recovered a lane later by a register-tiled batched twin that is proven bit-identical per row to router_gemv — a perf seam, not a numeric one (research/fast-router-20260802/).
The resulting contract, as written in SERVING.md: greedy serving is isolated-identical under concurrent load at defaults. Your output does not depend on your neighbors. Byte-identical alone or in a full batch — and gated on every model onboard, not assumed, via the same c=1-vs-c=16 replay that caught the defect (run-serve-gate.sh).
The exception, in the same breath
That contract has one documented boundary, and it belongs in this paragraph, not a footnote.
Serving primes prompts batched — prefill GEMMs at m equal to the prompt length. The historical oracle stream is tokenwise — decode kernels at m=1. Those are two different numeric configs by design, and on prompts where the model’s first output token is a near-tie, they can disagree. Measured on a six-model, 144-prompt sweep (prime-gate-coverage RESULTS.md): 10 of 144 first tokens flip, about 7%. Every flip sat at a tokenwise top1-top2 margin of 0.70 or less; the batched prime is bit-deterministic run-to-run; dense Q8_0 models — the fleet class — flipped 0 of 48. And in 8 of the 10 flips, a third numeric config sided with the batched prime, making the tokenwise “oracle” the outlier. This is config roulette on a near-tie, not a wrong path — but the consequences can be visible: on one probe prompt the batched config greedy-emits a newline and EOS at 2 tokens where the tokenwise stream writes 128. Within contract. Real.
The precise contract, then: identical numeric configuration in, identical bytes out — batch composition, neighbors, and concurrency can never touch your output. Cross-config identity on near-tie prompts is not promised; it is measured (~7%), bounded (margin ≤ 0.70), and escapable — MEMRA_PRIME_TOKENWISE=1 pins the tokenwise stream at prefill-throughput cost.
Why this is the property that matters
If you build evals or agents, this defect class should bother you more than a latency regression, because it is invisible to everything you normally monitor. Throughput unchanged. Loss unchanged. Every individual run deterministic. The only symptom is that results stop being a function of the input.
Concretely, determinism under load is what buys you:
Reproducible evals. If output depends on batch composition, your eval score depends on what else the harness had in flight — concurrency 8 versus 16 becomes a hidden hyperparameter, and a 0.5-point delta between two checkpoints may be routing noise. Byte-identical isolation removes the term entirely: rerun any single prompt from a 10,000-prompt sweep alone and get the sweep’s exact bytes.
Debuggable agent loops. An agent trace that went wrong at step 40 is only debuggable if replaying step 40 reproduces step 40. When co-arrivals shift expert routing, replaying yesterday’s failing request on today’s traffic gives you a different trajectory, and the bug report dissolves.
Auditable caching. A prefix cache is a claim that cached bytes equal recomputed bytes. Under the defect, a prefix primed in one batch and reused in another was a claim you couldn’t check. Under the contract, cached-versus-fresh identity is a gate (16/16 in the prompt-cache battery), not a hope.
None of this is a benchmark aesthetic. It is the difference between a serving stack you can reason about and one you can only observe. The uncomfortable part — and the reason the gate exists — is that the defect was caught by byte comparison during model onboarding, while a supported model carried it silently past its own gate. Determinism is not a property you have until proven otherwise. It is a property you gate, on every model, at every toolchain bump, with the exceptions written down next to the wins.
Receipts: findings.jsonl, the full concat-prime-exact lane, SERVING.md (“The isolation contract” and “First-token cross-config drift”), and the prime-gate coverage lane.