DeepSeek-V4.1-Flash on 2× NVIDIA DGX Spark (GB10, 128 GB unified) — vLLM TP2 Deployment

2-node DGX Spark setup serving DeepSeek-V4.1-Flash with vLLM tensor-parallel size 2 over ConnectX-7 RoCE, EXL3 2.9 bpw mul1 weights, and native NVMe-resident Engram tables.

  • 量化 EXL3 2.9 bpw (mul1)
  • 仓库 Mia-AiLab/DeepSeek-V4.1-Flash-EXL3-2.9bpw
  • 已核上下文262,144
  • 设备数2 × NVIDIA DGX Spark (GB10, 128 GB unified)

来源报告 https://github.com/MiaAI-Lab/DeepSeek-v4.1-Flash-EXL3-2x-DGX-Sparks

  • 31.6 tok/s (decode-tok-s)
  • 42.5 tok/s (decode-tok-s)
  • 42.8 tok/s (decode-tok-s)
  • 1054.7 tok/s (prefill-tok-s)
  • 872.6 tok/s (prefill-tok-s)

This deployment recipe is currently maintained in English because commands and technical instructions are source-aligned.

DeepSeek-V4.1-Flash on 2× DGX Spark (EXL3 2.9 bpw) is a working community deployment for DeepSeek-V4.1-Flash running on 2 × NVIDIA DGX Spark (GB10, 128 GB unified) using vLLM and EXL3 2.9 bpw (mul1). Verified up to 262,144 tokens context.

Measured by upstream maintainerworking

Tested Setup & Specifications

  • Model CheckpointDeepSeek-V4.1-Flash
  • Hardware SKUNVIDIA DGX Spark (GB10, 128 GB unified)
  • Device Count2 devices
  • InterconnectConnectX-7 RoCE (PTP link)
  • Runtime EnginevLLM (0.1.dev20904+g179dd0fa9)
  • Patches RequiredCustom kernel patch
  • Artifact FormatEXL3 · 2.9 bpw
  • Active Weights197.01 GiB
  • Tensor ParallelismTP = 2
  • Placement2 nodes, 1 GPU/node (GB10 unified)
  • Verified Context262,144 tokens
  • Last Verified2026-09-23

Memory & Storage Strategy

This setup relies on tiered storage and offloading strategies to fit larger weights and caches into physical memory:

  • Engram n-gram tables (shards 47+48) offloaded to NVME — Packed into per-rank hash-head shards on local NVMe via ./start.sh pack (or NFS share)

Tested & Reported Capabilities

  • ✓Text generation verified
  • ✓Clean model loading & initialisation
  • ✓Context scaling verified up to 262,144 tokens
  • ✓Function & Tool calling (Verified)
  • ✓Multimodal vision input (Verified)
  • ✓Multi-Token Prediction / Speculative draft (Verified)

Source-Backed Benchmarks

Measured performance on actual physical machines recorded by the upstream maintainer or community report. No linear extrapolations or simulated numbers.

MetricMeasured ValueContextConcurrencyDetails & Source
Decode Throughput31.6 tok/s—1 streamprose decode (temp 0, thinking off)prompt: 256 tokSingle stream decode with DSpark k=3 speculative decoding, TTFT 221 ms [source ↗]
Decode Throughput42.5 tok/s—2 streamsprose decode (aggregate)prompt: 256 tok2 concurrent streams aggregate (21.6 tok/s per stream), TTFT 347 ms [source ↗]
Decode Throughput42.8 tok/s—4 streamsprose decode (aggregate)4 concurrent streams aggregate [source ↗]
Prefill Throughput1,054.7 tok/s32,768 tokens1 streamprompt: 32,789 tok32k prompt prefill, TTFT 31.09 s [source ↗]
Prefill Throughput872.6 tok/s262,144 tokens1 streamprompt: 262,158 tok256k prompt prefill, TTFT 300.44 s [source ↗]

Quick Start & Deployment Commands

Condensed deployment procedure extracted directly from the upstream repository.

Prerequisites

  • 2× NVIDIA DGX Spark with ConnectX-7 point-to-point RoCE
  • Docker with NVIDIA Container Toolkit (sm_121a support)
  • ~440 GB free NVMe disk space for weights and local Engram shards

Review these commands, then run them yourself. This page does not run them. This is not the deployment brief.

1Configure environment
cp .env.example .env
2Start serving (head node)

Automatically fetches weights, pulls GHCR container image, configures NFS/ZFS, and serves API on :8888

./start.sh
3Pack Engram tables to local NVMe (optional)

Writes per-rank hash-head Engram shards onto each node's local NVMe (+25-50% prefill speed)

./start.sh pack
4Operational status and logs
./start.sh status && ./start.sh logs

Caveats & Operational Constraints

Open Calculator With This Exact Setup

Source & Provenance