guide · ai

Qwen3.8-Flash-Next on DGX Spark: Fast Only If the Kernels Load

NVFP4 weights plus MTP speculative decoding turn the DGX Spark into the strongest local path for Qwen3.8-Flash-Next, if the b12x kernels are actually loaded.

August 26, 2026 · By Alastair Fraser

A retro-futurist robot inspecting a sleek DGX Spark mini-workstation whose screen shows the GPU load meter pegged at full.

Qwen3.8-Flash-Next is the first open-weight release on the Qwen4 architecture: a 125B-parameter MoE that activates 6B per token, plus a 51B N-gram embedding table and a 4B Multi-Token Prediction (MTP) head — about 180B total on disk (HF model card). 48 layers, 36 Gated DeltaNet (linear attention) plus 12 Qwen Sparse Attention (QSA, micro-block indexed), 262K native context, extensible to 1M via YaRN.

On a single DGX Spark (DGX Spark — NVIDIA’s compact workstation with GB10 Blackwell chip, 128 GB unified CPU+GPU memory, $3–4K) with its GB10 Blackwell (Blackwell — NVIDIA’s 2025 GPU architecture, compute capability 10.0+; successor to Hopper H100) GPU and 128 GB unified memory, the right setup is NVFP4 weights + MTP speculative decoding + FP8 KV cache (NVFP4 — NVIDIA’s 4-bit floating-point micro-scaling format for Blackwell-class GPUs) served via vLLM. That combination is what makes Flash-Next feel like a different model than on other hardware. This guide walks through the operator-grade recipe, the b12x (sm_121 — Blackwell’s compute capability identifier for NVFP4 + b12x kernels) kernel foot-gun, and the realistic numbers.

Why NVFP4 + MTP is the sweet spot

Three levers converge on DGX Spark:

NVFP4 weights pack the 125B + 51B + 4B into ~110 GB at 4-bit class, fitting the 128 GB unified pool with room for KV cache. NVFP4 is the Blackwell-native format.
MTP is part of the checkpoint. The --speculative-config '{"method":"mtp","num_speculative_tokens":3}' flag in vLLM turns the head into a draft for speculative decoding (vLLM recipe).
FP8 KV cache halves attention-cache memory, which on this model is already small: 2 KV heads × 12 QSA layers ≈ 25 KB per token, an order of magnitude below the dense 27B’s cache (Atomic Chat).

Combine them and you get the architecture’s strongest path. The numbers from the dual-RTX-PRO-6000 run anchor what this combo can do: 123 tok/s baseline, 212–259 tok/s with MTP depending on reasoning_effort (Daniel Lougen, X, 2026-08-26). That is the upper bound for the architecture — do not extrapolate it to a single GPU, a non-Blackwell card, or a 3090. The Spark’s lower memory bandwidth puts a single-node number in the 70–100 tok/s range. Treat that as an estimate; Joey’s 2× DGX Spark first-look is pending verification.

The #1 foot-gun: b12x (sm_121) kernels

NVFP4 requires Blackwell-generation kernels — specifically b12x / sm_121, which the GB10 and GB300 use. Stock vLLM Docker images often miss them. If you skip this check, the server silently falls back to marlin W4A16 and you lose most of the NVFP4 throughput advantage.

Verify before serving:

python -c "
import torch
from vllm.utils.flashinfer import (
    has_flashinfer_b12x_gemm as gemm,
    has_flashinfer_b12x_moe as moe,
)
cap = torch.cuda.get_device_capability()
print('cap', cap, '| b12x gemm', gemm(), '| b12x moe', moe())
assert cap[0] == 12 and gemm() and moe(), \
    'b12x unavailable: serving would degrade to marlin W4A16'
"

If either flag returns False or cap[0] != 12, you do not have b12x kernels. Switch to the Unsloth Dynamic 3.0 GGUF path on llama.cpp instead — slower startup but portable. Set CUTE_DSL_ARCH=sm_121a in your environment for the FlashInfer/CuTe DSL build:

export CUTE_DSL_ARCH=sm_121a

The recipe (NVFP4 + MTP on DGX Spark)

This is the operator-grade vLLM command. It assumes the b12x check above passed and the NVFP4 weights are in place — the vLLM recipe page is the canonical reference for weight acquisition:

vllm serve nvidia/Qwen3.8-Flash-Next-NVFP4 \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.5 \
  --max-model-len 262144 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --async-scheduling \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --load-format fastsafetensors \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice

Three flags need explaining:

--gpu-memory-utilization 0.5 — the recipe default for UMA. Spark’s 128 GB is unified, not discrete. Raising this value causes overcommit and the 51B N-gram embedding table collides with the KV cache. Leave it at 0.5 unless you have measured headroom.
--speculative-config '{"method":"mtp","num_speculative_tokens":3}' — the MTP sweet spot. 1–2 gives no measurable speedup; 4+ collapses acceptance rate and you end up slower than the baseline.
--reasoning-parser qwen3 + --tool-call-parser qwen3_coder — enable_thinking defaults on (see below), so the parser strips <think>...</think> before returning the answer. The qwen3_coder parser lets the model emit structured tool calls during agent runs.

For vision workloads, append --limit-mm-per-prompt '{"image": 4, "video": 1}'. The vision encoder adds roughly 1 GB but lives comfortably in the remaining unified memory.

Reasoning-mode off-switch

Qwen3.8-Flash-Next ships in thinking mode by default: “Qwen3.8-Flash-Next models operate in thinking mode by default, generating thinking content signified by <think>...</think> before producing the final responses” (HF model card). Default reasoning_effort is xhigh.

For agent work, do not turn it off or lower effort. The vendor is explicit: “In multi-turn agentic tasks, lower reasoning effort does not always reduce overall task completion time. Although it may produce faster per-turn responses, it can also lead to insufficient analysis, more failures, and repeated retries, which may increase total latency and token consumption.” Lower effort is for short single-turn completions. For direct answers without the chain-of-thought block, pass chat_template_kwargs={"enable_thinking": False} in the request.

MTP sweet spot — and the llama.cpp gap

num_speculative_tokens=3 is the sweet spot for Qwen-family MTP heads. The MTP layer drafts tokens ahead, the verifier accepts or rejects. With n=3 you get a 1.7–2× throughput lift on Blackwell-class hardware (DJ Lougen: “[baseline] ~123 tok/s. With MTP: 212–259 tok/s … Minimum increase: 1.72x”). Raising n past 3–4 produces drafts the verifier rejects, and wasted compute drags throughput below baseline.

MTP support in llama.cpp is not yet verified for this model. The head is in the checkpoint; vLLM and SGLang support it; llama.cpp may not — check upstream issues before claiming it. On llama.cpp, expect no MTP speedup until upstream lands support.

What about that 212–259 tok/s number?

That figure is dual RTX PRO 6000 (96 GB each), NVFP4, with MTP n=3 (Daniel Lougen; confirmed in the NVIDIA dev forum thread). It is the architecture’s upper bound, not a single-GPU promise. It is not an Ampere or Ada Lovelace number — NVFP4 needs Blackwell. It is not the 24 GB RTX 3090 number; the UD-IQ1_S path on a 3090 + 96 GB system RAM is 3–8 tok/s with hybrid GPU/CPU offload (sibling Qwen3.8-Flash-Next RTX 3090 guide).

Don’t conflate the SKUs

The Qwen3.8 lineup has three open SKUs:

SKU	Params	Active	Notes
Qwen3.8-Flash-Next	125B + 51B + 4B	6B / token	This guide. MoE, hybrid GDN+QSA, MTP.
Qwen3.8-27B	27B dense	27B / token	Pick this on 24 GB cards.
Qwen3.8-Max	2.4T	API-only	No local weights. Don’t try to run it.

The hosted Qwen3.8-Flash on Qwen Cloud is the same checkpoint with 1M context and built-in tools — useful to compare local vs hosted behavior, but the open-weight Qwen3.8-Flash-Next is what you serve from this guide.

A useful framing from the Atomic Chat first-party guide: “On Qwen’s own launch table, Flash Next also comes out ahead of Claude Opus 4.6 Max on eight of the nine language benchmarks where both report scores. An Opus-level model now runs on a 128 GB workstation.” The same guide notes: “If your machine is a 24 GB card, the dense 27B remains the stronger pick for local coding; Flash Next is the better model when you have the memory for it.”

Pitfalls to surface in your runbook

b12x kernels missing — verify before claiming NVFP4. The 5-second check saves hours of “why is marlin so slow.”
--gpu-memory-utilization 0.5 on UMA — the recipe default. Don’t bump it; Spark memory is shared with CPU/OS.
N-gram embedding offloads to host — on UMA the 51B lookup is free; you don’t need 180 GB of discrete VRAM. 4-bit class fits; 6-bit (~165 GB) does not.
MTP n=3, not n=5 — start at 3. Acceptance collapses past 4.
Reasoning mode is on by default — xhigh is correct for agents. Don’t lower it.
Fake repos — only download from Qwen/Qwen3.8-Flash-Next, unsloth/Qwen3.8-Flash-Next-GGUF, or the Atomic Chat guide. Some pre-release HF mirrors were malicious.
Vendor benchmarks are inference, not third-party — every model card row uses the Claude Code harness at temperature=1.0, top_p=0.95, 256K context (HF model card).

When NOT to use NVFP4

NVFP4 is the strongest path on DGX Spark / GB10 and other Blackwell-class GPUs. It is not general-purpose. On RTX 3090 (Ampere, sm_86) or older Ada cards, switch to the Unsloth Dynamic GGUF ladder via llama.cpp — sibling guide. On a 24 GB card with limited system RAM, the dense 27B is still the better pick.

The DGX Spark is the sweet spot for this model: enough unified memory for the 4-bit class, b12x kernels for NVFP4, and enough compute for MTP. Verify the kernels, set --gpu-memory-utilization 0.5, leave MTP at 3, leave reasoning at xhigh, and you have the architecture’s strongest local path.

Done means

vllm serve is up — curl http://localhost:8000/v1/models returns 200 with the Flash-Next model id.
b12x-kernel probe returned cap=(12, ...), gemm()=True, moe()=True (see the foot-gun check).
--gpu-memory-utilization 0.5 is set; don’t bump it on UMA.
MTP is on with --speculative-config '{"method":"mtp", "num_speculative_tokens":3}'.
reasoning_effort is xhigh for agent work — do not lower it.