guide · ai

GLM-5.3-Flash on DGX Spark: The Verified 4x GB300 Recipe and the Single-GB10 Reality

The verified GLM-5.3-Flash recipe is SGLang on 4x GB300 (FP8 weights, FP8 KV, TRT-LLM DSA, NEXTN MTP): 1,870 tok/s aggregate. On a single DGX Spark expect far less. Weights are ~306 GiB.

August 26, 2026 · By Alastair Fraser

A retro-futurist robot beside an open compact workstation cube with glowing GPU core lights.

GLM-5.3-Flash on DGX Spark: SGLang, FP8 KV + MTP — and the single-GB10 reality

One-line job: set up GLM-5.3-Flash on Blackwell hardware with the only verified stack (SGLang), know which published numbers belong to a 4-node cluster rather than your desk, and run the reasoning-effort migration cleanly. Audience: DGX Spark / GB10 owners and anyone with multi-GPU Blackwell capacity. Not for: Mac or RTX 3090 readers — separate guides; none of the cluster numbers below apply to those boxes. Last verified: 2026-08-26

The problem

GLM-5.3-Flash is Zhipu AI’s open-weight 320B-total / 18B-active MoE with hybrid attention (MLA + DSA + KDA) and a native multi-token prediction head for speculative decoding. The only framework with a verified benchmark matrix at launch is SGLang, measured on a 4× GB300 node. The model ships as FP8 safetensors at roughly 306 GiB on disk — so before touching any command, decide whether your target even holds it.

On paper this sounds like a DGX Spark story: GB10 offers 128 GB unified memory and Blackwell kernels. The gap between those two sentences is this article.

What was actually verified (and on what hardware)

From the SGLang cookbook entry (2026-08-26):

“Measured on GB300: faster than BF16 + TileLang with about 1.8x the KV token capacity… adaptive MTP 5/1/6… 80 random requests at 1,024 input / 256 output tokens and concurrency 16 produced 1,870.82 aggregate output tok/s.”

Four facts hide in that quote, each load-bearing:

That number belongs to four GB300 GPUs serving one model. Divide nothing down to your Spark. A single GB10 has roughly an eighth of that silicon and must hold weights, KV cache, and activations inside its unified memory envelope.
“About 1.8× the KV token capacity” versus BF16 KV comes with a qualifier the headlines drop: “with GSM8K accuracy within noise of BF16” — at identical pool bytes. It is a cache-efficiency win, not a quality change.
MTP ran as NEXTN with an adaptive 5/1/6 profile accepting ~3 tokens per step. Strip any one of FP8-KV, TRT-LLM DSA backend, or speculative decoding and you are measuring a different configuration.
The claim class is [VENDOR/FRAMEWORK HARNESS — INFERENCE]. Nobody independent has rerun it yet.

The single-GB10 memory problem

Weights: ~306 GiB FP8. Your Spark’s unified memory ceiling: 128 GiB shared across CPU and GPU. Even if every scheduling trick worked perfectly, weights alone exceed physical capacity by more than 2×. There is no FP4 quantization of this checkpoint published by Z.ai, unlike their earlier GLM-5.2 hybrid-quant releases.

Honest options ranked:

Option	Verdict
Wait for community FP4/low-bit quants	Only path that could truly fit; none exist at verification time
Multi-GPU Blackwell server (not a Spark)	Where the verified recipe actually runs
Z.ai hosted API from a Spark-shaped desk	Works today, zero local footprint
KTransformers CPU-expert offload	Documented for big-RAM x86 + RTX 40/50-series (SM89/SM120) — not GB10, not macOS

Throughput reference points that ARE real

Three numbers worth internalizing so you can calibrate claims you see elsewhere:

Configuration	Figure	Source class
4× GB300, FP8 KV + TRT-LLM DSA + adaptive MTP	1,870.82 aggregate tok/s, mean TPOT 6.19 ms at concurrency 16	SGLang cookbook, framework-verified
Same rig vs BF16+TileLang baseline	3.2% higher aggregate, ~1.8× KV token capacity at identical pool bytes	SGLang cookbook
GLM-5.2 on a 3× GB10 cluster (prior model!)	~16.13 tok/s decode at 215K context, TP 3	NVIDIA dev forum

That third row is the sobering one: last-generation GLM across three Spark-class boards managed mid-teens tokens per second at long context. Inference-side improvements since then are real but incremental; until a sub-128-bit quant lands, your single GB10’s honest expectation for this model is “does not run,” not a specific tok/s figure.

Vendor’s own reference-harness disclosure puts GLM-5.3 at roughly 115 tok/s rescaling against competitor harnesses — a datacenter-serving figure, not an edge-device one.

If you bought a Spark specifically for frontier-open models like Qwen3.8-Flash-Next, note the contrast: that 125B/6B MoE fits comfortably in 128 GB unified memory with day-one NVFP4. GLM-5.3-Flash does not fit, full stop, until quantized builds appear.

Why the hybrid attention needs its own runtime code

Four mechanisms ship in one checkpoint, each new to inference engines:

MLA (multi-latent attention) — DeepSeek-style latent KV compression; mature support exists.
DSA (deepseek sparse attention) — a micro-block indexer that scores and selects top-k blocks before attention runs; needs engine-level integration, not just weight loading.
KDA (Kimi delta attention) — linear-recurrent states with a fixed-size pool; engines must allocate and schedule it as a first-class resource beside KV.
mHC (manifold-constrained hyper-connections) — residual-stream wiring from training; no serving cost but requires architecture-aware weight handling.

SGLang earned the verified-matrix spot by wiring all of these plus NEXTN speculative decoding together. That is why every recipe above names SGLang rather than whichever engine you already run — port quality, not partisanship.

The recipe where it DOES run (multi-GPU Blackwell)

For completeness — the verified low-latency shape you would migrate toward once capacity exists:

# SGLang container with GLM-5.3-Flash support (see cookbook for exact image tag)
python -m sglang.launch_server \
  --model zai-org/GLM-5.3-Flash \
  --tp-size <num_gpus> \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6 \
  --trust-remote-code --port 8000

Concurrency note straight from the cookbook: “Speculative decoding (NEXTN) is on — SGLang resets —max-running-requests to 48 when it isn’t set.” If you need different concurrency headroom, set it explicitly; do not let the default surprise you under load.

KDA state adds a second pool beside attention KV. At high concurrency the KDA pool can throttle before KV does — the cookbook documents --max-mamba-cache-size tuning for exactly this. Size both pools against your expected concurrency, not just token context.

Reasoning mode: the migration nobody escapes

This model family cannot turn thinking off:

“GLM-5.3 supports three thinking effort levels: low, high, and max. Disabling thinking is no longer supported by GLM-5.3.” — Z.ai blog, 2026-08-14

Migration rule for existing agent stacks: keep thinking.type at enabled, set reasoning_effort to low, verify responses, then raise effort deliberately to high/max where quality matters. Coding agents should live at max — vendor-recommended — while chat endpoints can afford low.

Pitfalls worth flagging

Lifting 1,870.82 tok/s into Spark marketing copy. Four-GPU number. Never yours.
Assuming vLLM parity. A recipe page exists for vLLM but the verified matrix, acceptance profiles, and pool-tuning guidance all come from SGLang first.
Confusing GLM-5.2 hybrid-quant work with this model. Different attention mix, different weight format; old quant recipes do not transfer.
Planning disk around model card size instead of ~306 GiB. Reserve 350+ GB free or the download stalls at 90%.
Leaving thinking.type: disabled payloads after migrating. Hard request failures.
Skimping on system RAM for multi-GPU hosts. Weight loading and offload staging want headroom beyond the VRAM sum; 1 TB-class hosts are the comfortable minimum for FP8.
Benchmarking before the KDA pool is sized. A mis-sized state pool throttles concurrency before KV does, and you will blame MTP acceptance for latency that is really scheduler queueing.

Done means

You can state which published numbers belong to 4×GB300 and stop quoting them for single-board plans.
Your storage plan reserves ≥350 GB for the FP8 checkpoint.
Any client you point at this model sends a thinking-mode-compatible payload (enabled + deliberate reasoning_effort).
You have the SGLang cookbook entry bookmarked as the canonical config source, ahead of blog summaries.
If you run a genuine Spark: you know today’s honest answer is hosted-API-or-wait-for-quants, and you watch Unsloth/Z.ai channels for a sub-128-bit build.