Conatus AI

Engagement summary. Qwen3-8B serving benchmark · RTX PRO 6000 Blackwell workstation · vLLM, SGLang and llama.cpp · measured 2026-08-25 · operator: Conatus AI.

This sample shows the evidence, analysis, and acceptance standard supplied with a serving benchmark engagement. Raw request records, machine-readable summaries, environment details, and reproduction scripts accompany the report. Download the raw evidence bundle (CSVs, summaries, scripts, manifest, 38 KB).

Executive verdict

Run the official Qwen3-8B-FP8 checkpoint on vLLM for a concurrent production workload on this hardware. It delivered 2,597.21 aggregate completion tokens per second at concurrency 32, the highest result in the test. The BF16 comparison also favored vLLM under load, so the recommendation does not depend only on quantization. FP8 raised vLLM's measured concurrency-32 throughput from 1,725.07 to 2,597.21 tok/s while reducing median end-to-end latency from 838.8 to 554.4 ms. For a lightly used service where reserved GPU memory matters more than batch throughput, llama.cpp remains a reasonable alternative because its measured allocation was substantially lower.

Method

One GPU in an RTX PRO 6000 Blackwell workstation: NVIDIA RTX PRO 6000 Blackwell Server Edition, 97,887 MiB memory, driver 590.48.01, compute capability 12.0, CUDA toolkit 13.1. Host: Ubuntu 24.04.4 LTS, Intel Xeon 6530P, 128 CPU cores, 503 GB RAM. Exact model revisions are pinned in the environment manifest inside the evidence bundle.

Stacks: vLLM 0.27.1 (torch 2.13.0+cu130), SGLang 0.5.9 (torch 2.9.1+cu128), llama.cpp at a pinned commit compiled for CUDA architecture 120. All stacks used a 4,096-token context per request. The workload is a fixed set of 20 prompts covering short and long questions, reasoning, code, summarization, translation, extraction, classification, writing, mathematics, and instruction following. Greedy decoding at temperature 0, streamed responses, 256-token cap. Concurrency 1 issued 20 requests; concurrency 32 issued 128 requests cycling the same set. Eight measured runs (two concurrency levels, three BF16 stacks, then vLLM FP8 on both workloads), three warmup requests each, server restarted between configurations.

Aggregate throughput is server-reported completion tokens divided by measured wall-clock time. TTFT runs from request start to first streamed token; end-to-end latency through completion; p50 and p99 from per-request samples. Peak VRAM is the maximum memory-used sample at 0.5-second intervals during a measured run.

Results

At concurrency 1 the three stacks are close on throughput and median latency; llama.cpp uses far less VRAM but its median TTFT is more than three times the vLLM result. Under concurrent load the stacks separate: vLLM leads on throughput and on median and p99 latency, SGLang is second, and llama.cpp trails materially on both.

The FP8 pass used the same workload and serving flags with the official FP8 checkpoint. At concurrency 32, measured throughput increased by 50.6 percent and median end-to-end latency decreased by 33.9 percent. Reported VRAM did not fall because vLLM preallocates most of the device in both runs; it should not be read as model weight size.

Quality regression check

Saved BF16 and FP8 concurrency-1 outputs were compared prompt by prompt: byte equality plus sequence similarity, with non-identical pairs reviewed side by side under a written rubric. Seven of 20 outputs were byte-identical. Nine differed only in wording or formatting and were judged equivalent. Three contained minor FP8 regressions: a repeated word in a poem, one list item that drifted off topic, and one questionable tool suggestion. One extraction prompt was ambiguous with two defensible readings. No case gave a wrong factual or numerical answer under FP8 that BF16 answered correctly. This supports FP8 for the tested prompt profile; it is not a general guarantee for other domains, longer contexts, or sampling settings.

Reproduction

The evidence bundle contains the benchmark client, the fixed workload, the quality comparator, an environment capture script, and a single entry point that reruns the complete benchmark and rewrites every CSV and summary in the results directory. One CSV and one summary JSON per measured run, with saved texts for the concurrency-1 quality comparison.

Acceptance terms

Deliverables are the raw CSVs, scripts, manifest, and this report. Acceptance criteria are the numeric tables above; if a re-run on the same hardware and config deviates by more than 10 percent on the headline metric, the balance is waived.

Back to the $349 benchmark