Conatus AI

Question. vLLM v0.25.1 with --moe-backend=flashinfer_b12x serving nvidia/Qwen3.6-35B-A3B-NVFP4 fails to start on 16 GB (RTX 5070 Ti) and 32 GB (RTX 5090) Blackwell cards: a torch.OutOfMemoryError is raised from FlashInfer's allocate_sm120_dynamic_workspace during profile_run, before the KV cache is sized. Is this a hard sm_120 incompatibility, or a memory-capacity ceiling? (Public issue: vllm-project/vllm#49476.)

Result. On a single RTX PRO 6000 Blackwell (sm_120, 96 GB) at TP=1, the identical image, model and flags start and serve. The path itself works on Blackwell workstation silicon; the smaller cards run out of room during profiling, not into an unsupported code path.

Environment

GPU        : 1x NVIDIA RTX PRO 6000 Blackwell, 96 GB, sm_120 (CC 12.0)
image      : vllm/vllm-openai:v0.25.1  (FlashInfer 0.6.13, PyTorch 2.11)
model      : nvidia/Qwen3.6-35B-A3B-NVFP4  (MoE + hybrid GDN, modelopt_mixed)
topology   : TP=1 / PP=1, single GPU

Command

vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --trust-remote-code \
  --moe-backend=flashinfer_b12x \
  --kv-cache-dtype=fp8 \
  --max-model-len=180224 \
  --max-num-batched-tokens=8192 \
  --max-num-seqs=8 \
  --gpu-memory-utilization=0.92

Measurement

What the large card exposes is how much the b12x path reserves before the KV cache is sized. The same config was then re-run with only --moe-backend changed to marlin (the backend that boots on the small cards), to isolate the backend's contribution on identical hardware and flags.

Same weights, same flags, same GPU, so the 11.47 GiB difference in what is left for the KV cache is the extra memory the b12x path reserves before the cache is sized: the sm_120 dynamic workspace, plus any activation it pulls in during profiling. That is consistent with a 32 GiB card failing during profile_run under b12x while marlin boots. The b12x reservation (weights plus this workspace, before any KV) is on the order of the whole small card, and --gpu-memory-utilization cannot help, because it only bounds what is allocated after profiling returns. The transient peak during b12x profile and warmup reached about 91.8 GiB here, briefly above the 0.92 utilization target, which is the moment a smaller card dies.

vLLM's own memory-profiler lines, verbatim:

flashinfer_b12x
  Available KV cache memory: 52.89 GiB
  GPU KV cache size: 5,358,120 tokens
  Maximum concurrency for 180,224 tokens per request: 29.73x
  init engine (profile, create kv cache, warmup model) took 262.72 s
  Application startup complete.

marlin (same image, model and flags)
  Available KV cache memory: 64.36 GiB
  GPU KV cache size: 6,518,438 tokens
  Maximum concurrency for 180,224 tokens per request: 36.17x
  init engine (profile, create kv cache, warmup model) took 168.37 s

Both runs reached Application startup complete. fp8 KV cache; correctness not audited beyond startup; single GPU, so no pipeline-parallel path exercised. Measured 2026-08-26 on vllm/vllm-openai:v0.25.1, operator Conatus AI. Related: vllm-project/vllm#49476.

---

To fingerprint your own environment and get its matrix key, run the open-source probe: uvx blackwell-doctor (source). It reports your Blackwell GPU, serving stack and the exact cell you are running, with no network calls.

This page is the shape of a single-cell serving verification: one public model and revision, one runtime, one quantization, one topology, one load point, measured on RTX PRO 6000 Blackwell and handed back with the exact command, environment and raw logs. If you have a Blackwell serving combination you need checked (does it start, does it stay correct, what does it cost), that fixed-scope check is $59, written delivery, no call.