Conatus AI

121.7 GiB is the Linux MemTotal we measured on one DGX Spark. The CUDA view on the same GB10 node reported 119.7 GiB. A sampler observed about 5.5 GiB in use with a Ray head and one GPU process running. For capacity planning, I use the lower CUDA total and round the remaining serving budget down to about 115 GiB.

The distinction matters because the advertised 128 GB is one shared physical pool. Resident weights, KV cache, compute buffers, CUDA contexts, Linux services and file page cache all consume it. free -m reports the Linux view, which is the right place to watch host pressure. The CUDA total is the ceiling used for accelerator allocation arithmetic.

Cgroup limits do not bound the shared pool

CUDA device allocations on GB10 are made through the driver and are not fully charged to the serving process cgroup. Another Spark owner measured a 10.7 GB GPU allocation while memory.current rose by only 377 MB in NVIDIA forum thread t/353752. systemd-run MemoryMax around docker run constrains the wrapper cgroup, while Docker creates a separate cgroup. Docker --memory constrains the container's accounted host heap. Neither setting caps driver allocations from unified memory.

Our two-node failure shows the consequence. During an attempt to reproduce vLLM issue 54919, MemAvailable on one node fell from 20,332,056 kB to 811,600 kB inside one 10 second sampler interval, then reached zero 44 seconds after the 811,600 kB sample. The worker was in nvidia_unlocked_ioctl with an rw-semaphore held. ray::DashboardA waited in the write path and nvidia-smi waited in the read path. The worker showed only 202 MiB of RSS while repeated earlyoom SIGKILLs had no effect. The process accounting therefore gave no useful picture of the device allocation, and a driver-path freeze had already made the worker uninterruptible.

That node had 16,383 MiB of swap. The first recovered journal excerpt showed 7.3 GiB in use, followed by about 6.2 GiB. Swap allowed host pages to move out while the driver consumed the shared pool, extending the thrash. Shrinking or disabling swap reduces that path without creating CUDA capacity.

Prevent and break an automatic restart loop

The author of NVIDIA forum thread t/382079 reported a single-node SGLang deployment with 64 GB of swap. Disabling Docker during the short responsive window had been unsuccessful. Our explanation is that the existing container was still running, its restart policy remained stored in hostconfig.json, and docker.socket could reactivate the daemon. live-restore: true adds another complication because the container can continue under containerd after the daemon stops and must be stopped separately.

Before testing a new memory configuration, arrange physical access or a smart plug configured to restore power after AC loss. DGX Spark has no BMC. Another poster reported using the smart-plug approach; the t/382079 author did not confirm the recovery sequence below.

From another machine, wait for SSH with a bounded, noninteractive probe:

until ssh -o ConnectTimeout=3 -o BatchMode=yes <user>@<spark> true; do sleep 0.5; done
ssh -t <user>@<spark> 'sudo docker update --restart=no <id> && sudo docker stop <id>'

Use the second command without -t only when passwordless sudo is configured. Change the restart policy before masking Docker. If the daemon cannot stay up, mask docker.service and docker.socket; with the daemon down, edit the existing container's hostconfig.json instead. Editing a Compose file alone does not change an existing container.

Projected single-node GGUF sizing

The following t/382088 example is arithmetic from our reply. It was never run. Its conditions are one GB10 Spark, Qwen3.8-Flash-Next UD-Q5_K_XL, -np 1, --cache-ram 0, an 80,000-token context, and vision enabled for a large image.

Projected component under those conditions	GiB
GGUF file	147.4
Lazy n-gram table removed from resident weights	-50.66
Host-side token embedding removed from CUDA weights	-0.63
CUDA weights after deductions	about 96.1
KV cache and model state at 80,000 tokens	about 2.6
Compute buffers	about 1.5
Multimodal projector	0.84
Vision compute for a large image	about 1 to 2
CUDA context	about 1.5
Projected pinned total	about 103 to 105

Against the rounded 115 GiB planning budget, this projection leaves 10 to 12 GiB. Long prompts still need several GiB of evictable page cache for on-demand table rows.

Lazy access is part of the configuration. Keep --lazy-mode auto; --lazy-mode off makes the 50.66 GiB table resident, and a GPU tensor override with -ot forces it onto the GPU as described in llama.cpp issue 28201. The optional --lazy-mode on-direct from PR 28136 removes most page-cache churn by reading requested rows directly.

UD-Q6_K_XL starts at 157.5 GiB. Applying the same deductions gives 157.5 - 50.66 - 0.63 = 106.21 GiB of projected CUDA weights. The source reply projects about 114 GiB pinned with the same runtime extras. A 115 GiB planning budget leaves about 1 GiB, which is too little for the lazy table's page cache.

Start with an allocation-free projection:

llama-fit-params -m <first-shard> -c 80000 -ngl 999 --fit-print on

Confirm that the load log reports lazy reads, then launch at -c 8192. Check free -m and /proc/pressure/memory after load and under the first large prompt. Advance to -c 80000 only after measuring the smaller run. For llama.cpp, --load-mode dio keeps bulk weight reads out of page cache while the lazy table retains its mmap path.

Two-node TP2 startup budgets

Our attempt to reproduce vLLM issue 54919 used two GB10 nodes, the official Qwen3.8-Flash-Next FP8 checkpoint, TP2 with expert parallelism, and image build 0.1.dev20073+g8e685d198. The public issue reports long-prefill decode starvation on a different community checkpoint at max-model-len 1000000. Our server never reached /health, so the attempt produced no latency result for that issue.

The failed launch used max-model-len 262144 and gpu-memory-utilization 0.80. Each node reported 87.42 GiB of model memory. The implied allocation budget was 95.76 GiB per 119.7 GiB node. At gpu-memory-utilization 0.84, related runs measured 92.88 to 96.15 GiB for weights plus non-torch memory and 4.07 GiB peak activation. Even the first ladder rung can fail under the lower memory fraction.

In an r2 run of the same official FP8 checkpoint at max-model-len 16384 and gpu-memory-utilization 0.84, one node measured 95.71 GiB consumed memory, 4.07 GiB peak activation and a 0.62 GiB KV budget. Other attempts on that node reported 0.17 to 3.45 GiB for KV. The observed range rounds to about 0 to 3.5 GiB of KV per node.

max-model-len does not set vLLM's KV block count. The memory fraction and --kv-cache-memory-bytes are the allocation levers. A context ladder beginning at 16384 remains a useful precaution because it limits the request shape after boot, while each rung must be restarted and measured. Before vLLM launch, clear disposable page cache with echo 3 > /proc/sys/vm/drop_caches.

The failed run's last sample before the collapse was about 20.3 GiB available. We use 25 GiB immediately before profiling as a provisional abort gate. No successful boot has validated that threshold. The old container watchdog required 15 consecutive low samples at 2 second intervals and recorded only sample=1/15; a 30 second persistence window could not react to the measured drop. A first-sample kill threshold, set from successful boots of the exact configuration and enabled only after weights load, is the safer host-heap response.

Both nodes should use the same host-heap guards. Our earlyoom configuration sends SIGTERM at 7 GiB and SIGKILL at 5 GiB of available memory. The internal gb10-oom-guard checks every 3 seconds and kills at less than 4 GiB. These guards can help while the offender remains killable. They cannot stop or recover the driver-path freeze described above.

Profiling needs separate headroom

In t/382099, another author's setup used GLM-5.3-Flash EXL3 at 4 bpw, DFlash2 with k=7, a 524k context, vLLM TP2 over RoCE, two ASUS Ascent GX10 systems and a community fork image. That author observed a second CUDA context fail on a 32 MB allocation when the shared pool was committed.

An in-process profiler uses the existing worker context, yet CUPTI device buffers and pinned host buffers still need an estimated few hundred MB. Restart the server with a lower --gpu-memory-utilization or a smaller --kv-cache-memory-bytes value for the capture. Restore the production allocation only after the trace has flushed and the server has restarted.

Preflight checklist

Item	Command or setting	Why
Read the Linux view	free -m	Confirm about 121.7 GiB total and measure margin in MiB
Project the GGUF	llama-fit-params ... -c 80000 ...	Estimate the single-node 80,000-token case without allocation
Preserve lazy reads	--lazy-mode auto, optionally on-direct	Keep the n-gram table out of resident weights
Stage llama.cpp	-c 8192, then -c 80000	Measure the smaller context first
Prepare vLLM cache	echo 3 > /proc/sys/vm/drop_caches	Remove disposable page cache before launch
Set the vLLM budget	--gpu-memory-utilization or --kv-cache-memory-bytes	These values determine KV allocation
Stage TP2 requests	Begin at max-model-len 16384	Limit the first tested request shape
Watch pressure	free -m and /proc/pressure/memory	Observe the shared Linux pool directly
Use an immediate guard	Kill on the first sample below the validated threshold	Avoid a 30 second persistence delay
Reduce swap	Shrink or disable it for the serving node	Limit prolonged host thrash
Remove automatic restart	docker update --restart=no <id> before testing	Keep a bad allocation from relaunching
Arrange recovery	Physical access or a restore-on-AC-loss smart plug	A Spark has no BMC
Profile with margin	Restart with a lower serving allocation	Reserve memory for profiler buffers

Sources: our measurements come from our attempt to reproduce vLLM issue 54919 on two GB10 nodes; the raw memory sampler, journal excerpt, kernel hung_task trace and watchdog script are in this gist. The GGUF arithmetic comes from our reply in NVIDIA forum thread t/382088. The boot-loop conditions were reported by the author of t/382079. The 32 MB context failure and profiling setup are the t/382099 author's observations. The cgroup accounting measurement is another Spark owner's result in t/353752.