Conatus AI

On sm_120 and sm_121, a model that will not serve often fails in one cell: one runtime version, one MoE or GEMM backend, one quantization, one topology. Change a single axis and the neighbouring cell behaves differently: the server starts, its output becomes correct or deterministic, or the KV cache gets 11 GiB more room.

Four pairs, three of them one axis apart

Backend. nvidia/Qwen3.6-35B-A3B-NVFP4 on vLLM 0.25.1, RTX PRO 6000 Blackwell (sm_120, 96 GB), TP=1, fp8 KV cache, --max-model-len=180224. With --moe-backend=flashinfer_b12x the engine reports 52.89 GiB of available KV cache. With only --moe-backend changed to marlin, 64.36 GiB. That 11.47 GiB is what the b12x path reserves on top of the marlin baseline before the KV cache is sized: the sm_120 dynamic workspace plus whatever activation it pulls in during profiling. The same command is reported to die inside profile_run on 16 GB and 32 GB Blackwell cards (vllm-project/vllm#49476) and starts on 96 GB: the b12x reservation, weights plus that workspace before any KV, is on the order of the whole small card. --gpu-memory-utilization does not help there, because the reservation happens before it applies. Full reproduction.

MoE finalize. nvidia/Qwen3-30B-A3B-NVFP4 on the vLLM nightly 0.28.1rc1.dev87, same card, TP=1, 8192 context, 4 generated tokens with top-20 logprobs at temperature 0. With FLASHINFER_CUTLASS auto-selected for the NVFP4 MoE, six identical requests returned six distinct logprob signature sets, differing at all four positions. With use_fused_finalize=False, one signature set in six, identical to 12 significant digits. That second arm also pinned --moe-backend flashinfer_cutlass, so the pair is not a single-variable A/B. Measurements and caveats.

Runtime commit. unsloth/Qwen3.8-27B-NVFP4 on SGLang, same card, TP=1, flashinfer backend. At commit c7e2c08d1, one commit before the fix in PR #35228, generation degenerates into a repeated phrase from the first token, content comes back empty, and the load log carries Parameter lm_head.weight_scale not found in params_dict. At main 07a9de25b the greedy spot-check is correct with zero scale warnings. The fix is in no release up to and including v0.5.18, so release wheels sit on the broken side of that commit. Signature and A/B.

One flag. unsloth/Qwen3.6-27B-NVFP4 on the vLLM nightly 0.26.1rc1.dev1214, GB10 DGX Spark (sm_121, 121.7 GiB unified), TP=1, 65,536 context. At --gpu-memory-utilization 0.60 the server starts and returns 12 of 12 needle probes across FULL-captured, eager and PIECEWISE graphs. Same box, same flags, at 0.92 vLLM sized 86.78 GiB of KV on the unified pool, the kernel OOM killer could not see driver-owned memory (engine RSS 31 MB against 152 GB of total_vm), and the machine needed a power cycle after 7 hours of thrash. The GB10 cell in full.

The axes that define a cell

The serving matrix records one cell per line with these fields: model_id, runtime, runtime_version, gpu, gpu_arch, vram_gb, quantization, kv_cache_dtype, tp, backend, context_len, concurrency, and then what happened in it: startup_status, correctness_status, agg_tokens_per_second, ttft_ms_p50, e2e_s_p99, available_kv_cache_gib, max_concurrency_x, peak_vram_gb, artifact_url, tested_at, note.

The first group identifies the tested configuration. A bug report containing only the model and the error text omits settings a maintainer needs to reproduce it.

Printing the cell

blackwell-doctor prints it. The package is not on PyPI, so run it from the repository:

uvx --from git+https://github.com/jahnclawdmonet/blackwell-doctor blackwell-doctor \
  --model nvidia/Qwen3.6-35B-A3B-NVFP4 --runtime vllm --quant nvfp4

Abridged output:

GPU        : NVIDIA RTX PRO 6000 Blackwell Server Edition
Arch       : sm_120 (Blackwell workstation/server) (cc 12.0)
VRAM       : 95.6 GiB
Driver     : 590.48.01
Stack      : python 3.12.3, vllm 0.27.1
Matrix key : nvidia/Qwen3.6-35B-A3B-NVFP4__vllm-0.27.1__nvfp4__tp1__sm12.0

It has no dependencies, makes no network calls, and does no GPU work beyond reading device properties. Run it inside the interpreter that has vLLM or SGLang installed. Run it anywhere else and it prints a note that no vLLM or SGLang import was found, so the cell is incomplete.

The GPU block comes from nvidia-smi --query-gpu=name,compute_cap,memory.total,driver_version, with torch consulted for the SM count and CUDA version when torch is importable and CUDA is available. On GB10 that query returns no dedicated VRAM, so the probe takes the total from torch when torch is present and CUDA is available, and from MemTotal in /proc/meminfo otherwise, labelling the result unified either way. A Spark has 121.7 GiB in that pool, and it is the same pool the driver allocates from.

The key itself is five fields joined by double underscores: model, runtime with its installed version, quantization, tp<N>, and sm<compute capability>, with every character outside A-Za-z0-9.+/- replaced by an underscore. Matching environments produce the same key, which you can quote in an issue.

What the key does not carry

--model and --quant are arguments you supply. The probe reads your GPU and your installed packages; it does not inspect a running server. An incorrect --model value goes into the key unchecked.

The key also omits six axes the matrix records: gpu, vram_gb, kv_cache_dtype, backend, context_len and concurrency. Two cases matter most. A 32 GB card and a 96 GB card with the same compute capability both produce sm12.0, so the key cannot tell them apart. The first comparison shows this for backends: flashinfer_b12x and marlin share model, runtime, quantization, topology and arch, so both backends produce the same key. Use the key to find candidate rows, then read the row's gpu, vram_gb, backend and kv_cache_dtype before concluding your cell was measured.

Same model, three runtimes

We used the same protocol for all three runtimes: Qwen/Qwen3-8B at bf16, TP=1, concurrency 32, on the same RTX PRO 6000 Blackwell.

vLLM's bf16 throughput is about four times llama.cpp's on that card.

Switching the same model to fp8 on vLLM 0.27.1 gives 2597 tok/s, roughly 1.5x over bf16. On GB10, under that box's own protocol, 451.3 tok/s at bf16 and 767.0 tok/s at fp8, a 1.70x ratio there.

Using it

Include the key in the bug report and check the matrix for matching rows. Where the failure signature is already known, the error index maps the literal error string to a reproduction or a confirmed diagnosis.

The matrix holds 21 measured cells at the time of writing, across vLLM, SGLang and llama.cpp on sm_120 and sm_121, known-broken rows included.

Sources

• blackwell-doctor: the probe, its output and the matrix key construction.
• blackwell-serving-matrix: the schema and the measured rows quoted above.
• Blackwell serving error index: the exact signatures, each mapped to a reproduction or a confirmed diagnosis.
• FlashInfer b12x sm_120 workspace, greedy nondeterminism at temperature=0 and DGX Spark bare-metal vLLM: conditions and limits for the four pairs.

---

For a missing configuration, we can run the measurement on the reference hardware and add a row: a fixed public model revision, runtime, quantization, topology and load point, returned with the exact command and raw logs. $59, written delivery, no call.