Two NVIDIA DGX Sparks, Three Open Models, and the Benchmark that Misled Us

Artificial intelligence is increasingly moving out of the cloud and onto hardware that organizations can own and operate themselves. That matters because modern AI coding tools can send enormous amounts of source code, documentation and other potentially sensitive data through a model while an engineer works. Until recently, running models powerful enough for serious software development required expensive data-centre infrastructure and specialist expertise. A new class of compact AI systems is changing that equation.

We wanted to find out whether two NVIDIA DGX Spark machines—small enough to sit beside a desk—could realistically replace cloud inference for day-to-day AI-assisted development, and, just as importantly, which measurements actually tell you whether such a system will work well in practice. What we found was that the benchmark numbers most commonly used to compare AI systems tell only a small part of the story.

A DGX Spark is a desktop machine, not a rack server. NVIDIA lists the Founders Edition at $4,699. Inside is a GB10 Grace-Blackwell chip with 128 GB of memory shared between CPU and GPU, which is the part that matters: there is no separate pool of VRAM to run out of, so a model that would need several data-center GPUs fits in something that sits under a desk and draws wall power. Two of them can be linked over a ConnectX-7 fabric.

That is the machine this article is about, and it is roughly the point where “run it locally” stops being a hobby answer and starts being a procurement question.

Many people have run frontier-class open models locally by now, gpt-oss-120b and Qwen3.8 among them. Very few have published numbers for what it actually does to a working day.

So, we put together a cluster in our lab to benchmark the models. Two NVIDIA DGX Spark nodes, a ConnectX-7 RoCE fabric between them, vLLM with tensor parallelism across both GPUs, and three openly licensed models that we could switch between without touching a service file. Then we measured it properly, with“vLLM bench” and real coding-agent sessions sampled from the /metrics endpoint, rather than repeating the marketing.

The result that mattered was not the one we set out to measure.

Figure 1. Six measurements from the two-node cluster. The bottom-right panel is the one that changed how we run it.

The Hardware Fact That Shapes Everything

Each Spark has one GB10 Grace-Blackwell GPU and 121 GiB of unified LPDDR5X shared between CPU and GPU. There is no discrete VRAM. Anything the OS, Ray, or the vLLM driver touches directly reduces what the GPU can allocate, and over-allocation surfaces as a kernel driver error instead of a clean CUDA exception.

The second fact is the interconnect. Each ConnectX-7 port negotiates a 200 Gb/s Ethernet link, but every function reports PCIe Gen5 ×4, about 126 Gbps usable. We measured ~104 Gbps with ib_send_bw. The link is PCIe-bound rather than fabric-bound, and jumbo frames changed nothing.

That matters because the obvious upgrade path, faster networking, is the wrong lever. Under sustained load our fabric sits roughly 88% idle.

Scaling is Real, but it is Not Linear

One first (huge) finding: Doubling the hardware does not double throughput. Single-stream, TP=2 delivers 21.11 output tok/s against 12.36 on one node: 1.71×, or 85.4% efficiency. At concurrency 32 that falls to 1.50×, or 75.1%. Tensor parallelism all-reduces activations on every layer of every token, so more in-flight tokens means more synchronization.

Latency improves alongside it. Mean time-to-first-token drops from 426.75 ms to 350.63 ms, and P99 from 504.30 ms to 375.36 ms.

The stronger argument for the second node is memory rather than speed. KV cache capacity goes from 2,040,430 tokens to 4,645,007, a 2.28× gain, because the driver overhead only lands on one node. And for genuinely large weights the second node stops being an optimisation and becomes a requirement.

Correctness Did Not Separate the Models

We keep gpt-oss-120b (Apache-2.0, 60.8 GiB, MXFP4), NVIDIA’s Nemotron 3.5 Lightning 30B-A3B (OpenMDW-1.1, 20.1 GiB, NVFP4) and Qwen3.8-27B (Apache-2.0, 20.4 GiB, NVFP4) cached on both nodes. The whole stack underneath is Apache-2.0 or MIT: vLLM, Ray, PyTorch, FlashInfer, Transformers, Triton. Nothing about this cluster is locked.

We ran the same five prompts against all three at temperature 0 with a shared 4,000-token budget, then extracted the generated code and executed it rather than eyeballing it. All three passed everything. No empty answers, no wrong output, and on a 30-task machine-verified evaluation all three scored 30/30. That is a ceiling effect and not proof of equivalence, and it is worth stating plainly because the interesting differences are elsewhere.

Worth one caveat on method: three bugs in our own checker surfaced during that run, including one that counted the “100” in a header line as a prime. All three models were initially marked as failing on artifacts of the harness. Anyone comparing model quality should execute their checks against known-good output before trusting a red result.

What Did Separate Them Was Style

The models spend radically different amounts of effort to reach the same answer. Measured in characters of hidden reasoning against characters of visible answer, gpt-oss reasons least and answers most: 266 characters of thinking produced a 3,508-character refactor. Nemotron is the heaviest reasoner by a wide margin, with 11,140 characters of thinking for one sieve function and 1,629 characters of reasoning to answer a one-sentence factual question. Qwen is the terse one, with the shortest answers throughout: 233 characters for a Node.js script against gpt-oss’s 1,606.

Figure 2. Same five prompts, same budget, three very different working styles.

Then there is the one behavioral difference that would actually change a deployment decision. Given a deliberately contradictory instruction, “write a Python script to produce prime numbers in Node.js”, Qwen and Nemotron both flagged the mismatch and asked which was meant. gpt-oss did not. It silently resolved the ambiguity by writing a Python script that generates a Node.js program.

That reading is defensible, even clever. It is also an assumption made without telling anyone. In an agent loop, where a wrong assumption compounds across turns and nobody reads the intermediate steps, the two models that surface the ambiguity are the safer default. This is the kind of property no leaderboard reports.

The Measurement That Changed Our Default

Synthetic benchmarks describe what a cluster can do. They do not describe what an agent does, which is what we really care to know.

One Claude Code session against Qwen moved 252,173 prompt tokens and produced 2,012 output tokens, a ratio of roughly 170:1. Peak KV cache usage was 0.00%. Coding agents are prefill-bound rather than decode-bound, so generation throughput (the number every benchmark reports) is close to irrelevant for this workload.

Run the identical task against Nemotron and wall time falls from 163 seconds to 19, about 8.6×. That is wider than the 5× ratio of raw generation rates, because the MoE prefills faster too. Nemotron is now our default.

Three concurrent sessions cost nothing measurable, since agent sessions idle in each other’s gaps. At six, per-session time rises 1.5 to 1.7×.

What This Actually Means

Two Sparks, a fabric that is mostly idle, and an entirely open stack will comfortably serve three or four engineers doing real agentic work, with every token staying on hardware you control.

That last part is the point. For code that must be verifiable, where provenance is a compliance requirement rather than a preference, the ability to run capable models without shipping your source to anyone else’s inference endpoint is an architectural precondition rather than a cost optimization.

The performance is now good enough that it is no longer the reason to say no.