Conatus AI

Symptom

Identical requests at temperature=0 produce different completions or logprobs. In vLLM #54521, bankjaneo reported this on Qwen/Qwen3.8-Flash-Next-FP8 across a DGX Spark GB10 pair (sm_121), TP=2. The initial sweep used indexer_budget=8192; the corrected follow-up located onset near the stock budget of 2048 and tied it to prompt prefill length. Generating past the budget from a short prompt stayed reproducible in that test. A strict threshold shared by every configuration is not established.

sudodrew later reported the same symptom on RTX PRO 6000 Blackwell Max-Q Workstation Edition (sm_120), TP=2, with the same checkpoint and indexer_budget=2048. Approximately 509-token prompts gave 1 distinct output in 6 requests; approximately 4,009-token prompts gave 6 in 6. These are sudodrew's serving measurements, separate from our dense-kernel tests below.

davidcanar's final report on gfx1151 / ROCm with GLM-5.3-Flash separated below-budget MoE router divergence from above-index_topk attention divergence. Earlier claims that the indexer was excluded applied only below its budget. The same report restates two earlier retractions: a prefix-cache exclusion that had not been controlled for, and a claimed --max-num-seqs 1 crash trigger that did not reproduce on retest.

Reproducibility and answer quality are separate checks. bankjaneo corrected the original interpretation: deterministic top-k did not fix the Thai corruption that motivated the issue. Do not treat an identical completion as a quality result.

What was measured on sm_120 by us

We measured dense projections on an RTX PRO 6000 Blackwell Server Edition, x86 Linux, driver 590.48.01, image vllm/vllm-openai:v0.28.0, torch 2.13.0+cu130, sm_120. One GPU, no model loaded, no network. Shapes, the M set and the M-invariance harness are jschmied's gemm_m_invariance.py; we ran it unmodified and rebuilt the scale layouts for the correction. Two questions are separate: whether row 0 changes with M (answered by the layout correction) and whether repeated calls at fixed M change at all (the fixed-M repeats).

Corrected M-invariance results, row 0 against the same arm at M=1. Tested M: 1, 2, 3, 4, 8, 9, 16, 32, 64, 128, 256, 1024, 4096. Differences below are measured in bf16 ulps at the magnitude of max|row 0|.

Correction. Our initial blockwise FP8 row used row-major activation scales [m, K/128] and row-major weight scales [K/128, N/128]. The sm120 kernel deduces M-major activation scales and K-major weight scales from the shape; it does not read their strides. That test computed an unintended GEMM. With the production layouts, the apparent M-dependence went away. The hand-built corrected arm matched a float64 reference to within half a bf16 ulp. The quantizer arm used UE8M0 scale values; its layout matched production, but its values differed from the hand-built arm. The per-channel and bf16 conclusions were unchanged. The bf16 q_proj islands remain unexplained: we did not inspect cuBLAS kernel selection, as stated in our follow-up.

Fixed-M repeat hashing

We held inputs and M fixed and hashed the full output matrix. M=740 was a dense-projection chunk size reported in the ROCm investigation; M=5571 was retained as a large-M point, although that prompt was chunked in the server. These synthetic calls do not recreate that server execution.

The blockwise cells used the incorrect layout described above: their repeat identity still holds, but only for that unintended GEMM. These counts are not fixed-M repeat evidence for the corrected production layout. The experiment used synthetic inputs, a single stream and an otherwise idle GPU, with no scheduler or collective. It did not exercise MoE grouped GEMM, attention or the ROCm stack. Matching launches also do not isolate workspace addresses, since the caching allocator can replay the same allocation sequence.

In the posted analysis, 30 calls per cell give only about a 10% upper divergence-rate bound at 95% confidence. Pooling 1080 calls gives a bound near 0.3% at 95%, conditional on a common rate across cells, including the unintended blockwise cells. The result constrains frequent output variation in these tested calls; it does not establish kernel structure or end-to-end determinism.

Our fused-finalize reproduction

For #54945, we measured nvidia/Qwen3-30B-A3B-NVFP4 on an RTX PRO 6000 Blackwell (sm_120, 96 GB), TP=1, FlashInfer 0.6.17, image vllm/vllm-openai:nightly-6d4562c59b97b4e35d459ff9389e71b6fe4995de. The runs used eager execution, prefix caching off, --max-model-len 8192 --max-num-seqs 16 --max-num-batched-tokens 4096 --gpu-memory-utilization 0.85. Each arm ran jschmied's probe unchanged: a single server start and 6 requests, generating 4 tokens with top-20 logprobs at temperature 0.

The patched arm also pinned --moe-backend flashinfer_cutlass. Both logs name the same backend, but the backend pin changed the autotune cache hash. This was not a single-variable A/B. No emulation control or throughput measurement was completed. These signatures hash top-logprob values formatted to 12 significant digits, rather than the full internal logit tensor.

What the thread established

Indexer selection above index_topk: persistent_topk

bankjaneo localised the mechanism on sm_121 first: the controlling variable is tie density at the k-th threshold, persistent_topk breaks ties non-deterministically, and qsa_mqa_paged and qsa_sparse_paged_attention were deterministic in that test. davidcanar reported that, after the router fix, identical inputs diverged at the sparse attention output above index_topk=2048. The report links PR #55122 for deterministic persistent_topk; testing that PR on gfx1151 was still pending in the comment. jschmied measured a further reduction in above-budget variation on sm_121 with deterministic top-k added to non-fused finalize. Other effects remained until the offload fix described below.

MoE grouped_topk Python fallback on ROCm

davidcanar reported call-to-call order changes in torch.topk(..., sorted=False) above 256 experts on gfx1151, ROCm 10.0, torch 2.11. Exact boundary ties could also change the selected set. The proposed fix, PR #55514, uses stable descending sorting in the grouped_topk Python fallback. It restored below-budget repeatability on that GLM-5.3-Flash stack. jschmied scoped this separately: Qwen3.8-Flash-Next selects FusedTopKRouter and CUDA topk_softmax, so this fallback fix does not explain that model's sm_121 results.

PLE CPU-offload semaphore on GB10

jschmied reported that CUDA-graph capture left the offload semaphore a step ahead, causing each forward to consume the previous step's PLE output. The comment points to #53899 and fork PR #13. With this fix, deterministic top-k and non-fused finalize, sequential tests at 1,460, 1,999 and 5,960 prompt tokens had 0 top-1 flips, 0.000 spread and 1 distinct 64-token completion in 8 requests. Concurrent batches retained 0, 416 and 665 flips respectively, matching the CUDA-graph-off run. The report distinguishes the remaining batch-shape sensitivity from the semaphore defect.

FlashInfer CUTLASS NVFP4 MoE fused finalize

jschmied reported in #54945 that the fused finalize reduces expert outputs with atomics, and that passing use_fused_finalize=False made the tested logprob signatures repeatable on GB10. The later regression comment retains that toggle alongside the indexer and offload fixes. The original report proposes exposing the argument in vLLM; it does not supply a fix PR. Our separate sm_120 reproduction and its backend-pin caveat are recorded above.

Checks you can run

Hold M and inputs fixed before changing batch shape

Use the fixed-M script and run records in the dense GEMM evidence bundle. Its hashing core is below. Reuse the same input tensors for each call, retain per-cell digests and compare separately launched processes with the same seed. For blockwise FP8, apply the documented scale-layout correction before interpreting it as a production GEMM check; the published original repeat script contains that mistake.

import hashlib
import torch

def digest(t):
    t = t.detach().cpu().contiguous()
    return hashlib.sha256(t.view(torch.int16).numpy().tobytes()).hexdigest()[:12]

# run(m) is the selected GEMM with fixed inputs; m stays fixed.
digests = [digest(run(m)) for _ in range(10)]
print(len(set(digests)))

More than one digest locates variation in the tested operation. Matching digests constrain that cell only. Then vary M separately to test batch-shape dependence. The integer view preserves bf16 output bits for hashing; direct NumPy conversion of bf16 failed in the initial script attempt, which produced no usable data.

Compare the same prompt position across requests

Follow jschmied's position-resolved protocol: prefix cache off, no speculative decoding, temperature 0, prompt_logprobs=5, 8 sequential and 8 concurrent identical requests, with prompts below, near and above the indexer budget. Compare forced-token logprob spread, top-1 agreement and top-k overlap at each position, plus generated-token hashes. A sequential difference and a difference caused only by concurrent batching call for separate investigation. A cold-first-request difference should also be separated from warm repeats.

ZC502 published vllm-position-parity for repeat stability, cross-arm parity and schedule sensitivity. ZC502 validated its schema and analysis against synthetic fixtures; the live collector had not been validated on sm_121 hardware in that report. Its measurements do not assign root causes or significance thresholds.

Test use_fused_finalize=False on the selected MoE path

In the build described by jschmied, pass use_fused_finalize=False to flashinfer_cutlass_fused_moe in flashinfer_cutlass_moe.py. Keep the backend pin and other settings identical between arms, verify the selected backend in both logs and compare repeat signatures. If variation disappears, the toggle separates the tested arms; if it remains, continue with the indexer, offload and scheduling checks.

jschmied also reported an autotune-cache collision on FlashInfer 0.6.17 when changing finalize mode. Use a fresh VLLM_FLASHINFER_AUTOTUNE_CACHE_DIR for each arm there; the report identifies the cache-key fix from v0.6.18rc2. sudodrew checked vllm/envs.py at main and v0.29.0rc4: VLLM_QSA_EXACT_TOPK does not exist at those references. There is no env-level escape hatch on a stock build.

Sources

• vLLM #54521: bankjaneo's report; corrections and attributed measurements linked at each finding.
• Our dense GEMM bundle: M-invariance, fixed-M repeats, corrected scale layouts, scripts and run records.
• vLLM #54945: jschmied's fused-finalize diagnosis, probe and cache caveat.
• Our sm_120 fused-finalize bundle: raw responses, server logs and patch.

---

To fingerprint your own environment and get its matrix key, run the open-source probe: uvx --from git+https://github.com/jahnclawdmonet/blackwell-doctor blackwell-doctor (source). It reports your Blackwell GPU, serving stack and the exact cell you are running, with no network calls.

A single-cell serving verification is one public model and revision, one runtime, one quantization, one topology, one load point, measured on RTX PRO 6000 Blackwell and handed back with the exact command, environment and raw logs. If you have a Blackwell serving combination you need checked (does it start, does it stay correct, what does it cost), that fixed-scope check is $59, written delivery, no call.