Run a private remote AI assistant on DGX Spark

Individual experiment · 12 September 2026

Two fast single-request runs produced complete HTML; neither solution worked.

A fresh Pi 0.85.1 experiment sent the same hardware-store plank-helper prompt to two resident models on one DGX Spark, sequentially, with one active request and low reasoning. Qwen 3.8 Flash Next NVFP4 with vLLM and MTP decoded at 34.538774 tok/s. Pi observed 34.411772 generated tok/s end to end: 10,198 generated tokens, including 4,434 reported reasoning tokens, in 296.352075 seconds from 459 input tokens. The response contained a complete HTML document, but a strict-mode runtime error stopped the cut list and diagram.

Muse Glimmer 30B NVFP4 with vLLM and DFlash decoded at 26.689318 tok/s. Pi observed 26.536490 generated tok/s end to end: 3,574 generated tokens in 134.682468 seconds from 441 input tokens. Muse emitted reasoning text, but its counter incorrectly reported zero reasoning tokens, so no reasoning-token count is claimed. Its response contained a complete HTML document, but an extra parenthesis caused a syntax error and disabled the calculator.

Both requests used a 500-second response limit, temperature 0.7, top_p 0.95, top_k 20, presence_penalty 0.25, repetition_penalty 1.1 and seed 42. min_p was omitted because the installed speculative-decoding paths rejected a nonzero value. The server metrics recorded uncached prefill of 459 tokens in 0.74378188 seconds for Qwen and 441 tokens in 0.509927573 seconds for Muse. The simulator derives about 617.116405 and 864.828700 prefill tok/s from those counters. These are individual experimental measurements, not concurrency results, recommended settings or successful coding solutions. The settings were discontinued after the experiment. One stochastic sample per configuration cannot establish a performance or quality effect.

Spark owner tests · checked 15 August 2026

Owner tests report decode in the 20s and 30s on one Spark.

Meta’s Apache-2.0 Muse Glimmer 30B is a dense local-agent model with tool use, coding and optional image input. Its official GGUF release provides 16.76 GB and 19.65 GB language-model builds, plus a separate 1.40 GB perception encoder and optional 1.63 GB DFlash drafter. That leaves ample room inside a 128 GB Spark-class system for runtime and a useful context budget, although the advertised 131,072+ model limit is not a promise that the maximum context will be fast or memory-efficient.

Early owner runs provide a useful speed range, not a controlled benchmark. On one DGX Spark, a llama.cpp run reported about 10.5 tok/s conventional decode and 36-38 tok/s with the official DFlash drafter at 15 speculative tokens; a separate vLLM run rose from 5-8 tok/s to about 23 tok/s at the same setting. Workload, draft acceptance, quant, context depth and engine build can move those numbers.

The llama.cpp test also reported about 700 tok/s prefill at short context, about 390 tok/s deep into an 832K-token prompt, and 3/3 needle retrieval at 97K, 188K, 415K and 832K after an 8× YaRN and metadata override. Treat that as an extended-context experiment rather than a new guaranteed model limit. Its two-Spark RPC layer split slowed decode to 25-28 tok/s, so a model this size is better run as one instance per Spark unless a measured workload proves otherwise. Pin the exact engine revision and rerun your own tool-call, recovery, context and latency harness.

Vendor video benchmark · checked 15 August 2026

The vendor reports 3.92× faster generation: about three minutes for five seconds.

NVIDIA’s SANA team ran the 33B dense MiniMax H3 audio-video generator on one DGX Spark at 832×480, 24 fps, 124 frames and 50 denoising steps. Sol Engine reduced end-to-end wall time from 710.6 to 181.3 seconds, a reported 3.92× speedup. The full recipe combines kernel work with approximate Sol-Attn sparse attention and cross-step caching, so the accompanying near-lossless quality assessment is part of the vendor experiment, not proof that every prompt is unchanged. NVIDIA’s separate GeForce RTX 5090 result uses 1344×768 and must not be treated as an RTX Spark benchmark or compared by raw time.

The ordinary BF16 FL2VA package is about 134.2 GiB before activations, so NVIDIA’s GB10 runtime uses a pruned FP8 DiT, FP8 Qwen3-VL conditioner and the released VAEs, with one resident model process. MiniMax’s downloadable local system produces 768p H3-Base output; the recommended Context-IR preprocessing and Regenerate-2K stage remain hosted. There is also a procurement-level license gate: the community license excludes the EU, UK, United States and South Korea, with a separate application route for those territories.

Practitioner handbook · checked 30 September 2026

EXO Labs' DGX Spark Handbook, written by 0xSero, maps current community recipes from one Spark to four. Its practical sequence is useful: establish remote access with NVIDIA Sync or SSH, confirm a local model works, then move to a pinned vLLM or SGLang recipe and connect your existing agent or Open WebUI to its OpenAI-compatible endpoint.

Evidence boundary: the handbook reports practitioner runs, including workload-dependent speculative decoding. Coding, structured output and prose can have different draft acceptance. Compare time to first token, sustained per-user decode, aggregate throughput, output correctness and whole-system power at the same context and concurrency. A fast local stream does not establish cloud-model quality parity or a purchase payback.

Four-node rings are a community option. SparkRing documents GB10 pairs and four-node rings using custom RDMA collectives and patched NCCL. It labels the stack alpha and asks operators to pin an immutable commit. Check each profile's model and draft licences; its public BF16 DFlash2 checkpoint is restricted to research and evaluation under CC BY-NC-ND 4.0.

Community implementation · checked 30 September 2026

remote-dspark moves the DSpark speculative draft model from a GB10 cluster to a separate networked GPU. The maintainer reports about 2.8 GiB freed per node for GLM-5.3 Int4-Int8Mix on four GB10 systems with an RTX 3090 drafter. That creates KV-cache headroom; it is not a measured speedup or a general memory guarantee.

The project supplies opt-in mods for a vLLM 0.29-based stack, a separate x86_64/sm86 draft-server image, and TCP/ZMQ or RDMA transports. The documented smaller-GPU option uses FP8 shared embedding/output-head tensors and a modest draft cache to fit in 10-12 GB. RDMA requires verbs devices on both ends; transport support does not establish GPUDirect RDMA support for GB10 CUDA allocations.

Match the target's shared tensors, draft checkpoint, speculative-token count and context budget. Start the draft service before the cluster. Remote vision-encoder offload is suggested but not shipped.

Keep the draft service on a trusted, restricted network. Pin the code and image, then compare against local drafting at identical prompts, sampling and concurrency. Record memory, latency, acceptance, failures and the extra GPU's power.

Community recipes · checked 23 August 2026

Full-model recipes depend on pinned custom stacks.

A July practitioner recipe serves the full GLM-4.7 355B Salyut1 NVFP4 checkpoint, about 188 GB across 41 shards, on two DGX Sparks over the built-in ConnectX-7 fabric. The author reports about 17.5 tok/s with vLLM TP=2, CUDA graphs enabled and a configured 65,536-token ceiling. The setup passed the author's coherent-completion and tool-call checks plus an exact needle-recall check around 21K tokens; that is not a full-64K quality validation. It is not a stock install: the recipe uses a development vLLM image, an open loader-fix pull request, a no-Ray launch path, explicit GB10 NCCL handling, capped batch settings and 0.90 memory utilization. The author also reports about 9.5 minutes to load the checkpoint.

An August Level1Techs report measures the official DeepSeek-V4-Flash-0731 304B checkpoint on two Sparks with TP=2 over dual 200 Gb/s RoCE. Here, NVFP4 describes the KV cache rather than the model weights. On a custom vLLM fork at the model's calibrated 1,048,576-token ceiling, with six sequence slots, five speculative tokens, warm kernels and non-streamed accounting, the author measured 84.1 tok/s peak, 67.5 mean decode, 219.2-226.9 aggregate tok/s at concurrency six and about 2,690 prefill tok/s at 100K. A separate 40-minute mixed-temperature, mixed-budget soak at concurrency four produced 87.2 aggregate and 21.8 per stream across 532 requests without a request error.

Do not turn the RTX comparison into a quality ranking. The forum's temperature-zero diagnostic found stack-specific counting and arithmetic differences, but its expanded artifacts were not publicly downloadable when checked. DeepSeek recommends temperature 1.0 and top-p 0.95 for agentic local use. The public recipe also changes quickly and contains results for both the official 0731 checkpoint and an earlier preview. Keep the checkpoint, commit, container, patches, prompt, sampling, warm-up, stream mode and concurrency attached to every number.

Native NVFP4 hardware · speculative capacity

Blackwell FP4 hardware is present; the serving kernel remains a gate.

NVFP4 is promising here because it can shrink weights and reduce memory traffic while preserving more quality than a crude four-bit conversion. For planning, reserving 20-25% of Spark's 128 GB and assuming a mixed checkpoint at roughly 5.0-5.2 bits per parameter gives an estimated 150-165B parameter single-node fit. That is below NVIDIA's “up to 200B” capacity ceiling because this estimate leaves useful room for the runtime and cache.

Two linked Sparks provide a speculative 295-330B parameter planning band with the same reserve. NVIDIA has demonstrated Qwen-235B in NVFP4 on two systems and markets an up-to-405B model ceiling, but neither statement guarantees your model, context, kernel or interactive speed.