Qwen3.8-Flash-Next on an RTX 3090: realistic 3-8 tok/s and why the dense 27B is faster

This is a recipe article for Qwen3.8-Flash-Next on one RTX 3090 (24 GB VRAM) paired with 96 GB of system RAM. There is no published Flash-Next benchmark on a 3090 at the time of writing. The honest ceiling is 3-8 tok/s with hybrid GPU/CPU offload — split weights between the 24 GB VRAM and system RAM, streaming experts on demand on UD-IQ1_S, and the dense Qwen3.8-27B will be 3-10x faster on the same hardware. If your goal is “fastest local Qwen3.8 on a 3090,” you almost certainly want the 27B instead.

This guide exists because people will still try it. The 3090 + 96 GB combo is the only consumer-class setup that can hold the Flash-Next 1-bit class at all, and the architectural story is interesting enough to be worth documenting honestly.

What Qwen3.8-Flash-Next actually is

Qwen3.8-Flash-Next is the first open-weight release on Alibaba’s Qwen4 architecture, dropped 2026-08-26. Per the Hugging Face model card: a 125B-parameter MoE that activates 6B per token, plus a 51B N-gram embedding table (bigram/trigram lookup at layer 2), plus a 4B Multi-Token Prediction head — on-disk total ~180B parameters.

The 3090 + 96 GB combo is viable because the N-gram table is offloadable to host memory:

“Embeddings provide a unique axis for parameter scaling that requires less computation and is more amenable to offloading than Mixture-of-Experts (MoE). By indexing with short n-grams, this approach makes parameter scaling highly efficient for memory-constrained accelerators without sacrificing quality.” — Qwen Team, HF model card, 2026-08-26.

You do not need 180 GB of VRAM. The N-gram table lives in system RAM and prefetches asynchronously; only the routed experts (the per-layer learned sub-networks activated by the MoE router; only some are active per token), shared experts, and the active 6B worth of attention need to hit VRAM on the hot path.

The 48-layer architecture mixes two attention types: 36 use Gated DeltaNet (linear attention, fixed-state, no cache growth) and 12 use Qwen Sparse Attention (2 KV heads). Per-token cache is roughly 25 KB — an order of magnitude smaller than the dense 27B at the same context.

Memory math on a 3090 + 96 GB

Per Unsloth coverage, UD-IQ1_S is 72.5 GB on disk — the floor for a Flash-Next GGUF that loads at all in this RAM tier.

Tier	Quant	Size	Status on 3090 + 96 GB
1-bit	UD-IQ1_S	72.5 GB	The realistic ceiling. Fits in 96 GB system RAM with the active experts on the 24 GB card. Quality drop is real.
2-bit	UD-Q2_K	~88 GB	Tight headroom. Long-context will OOM.
4-bit	UD-Q4_K_XL	~110 GB	Does not fit in 96 GB system RAM. Will heavy-swap.
6-bit	UD-Q6_K_XL	~165 GB	Not viable.

UD-IQ1_S at 72.5 GB is the path that loads and runs without page-thrashing. Do not confuse the 4-bit GGUF (~110 GB) with the BF16 reference checkpoint (~360 GB) — only the GGUF is runnable on consumer hardware.

The honest speed ceiling

No published RTX 3090 number exists for Qwen3.8-Flash-Next. The closest reference is the dual RTX PRO 6000 NVFP4 + MTP result of 212-259 tok/s per Daniel Lougen’s first-look thread:

“First numbers of Qwen3.8 Flash Next NVFP4 on dual RTX PRO 6000. Baseline: ~123 tok/s. With MTP: 212-259 tok/s (depending on reasoning effort) Minimum increase: 1.72x.” — Daniel Lougen, X/Twitter, 2026-08-26.

That number is dual RTX PRO 6000, NVFP4, with MTP n=3 — the architecture’s upper bound on a Blackwell-class workstation, not a number any single GPU will hit. It is especially not a number the 3090 will hit: NVFP4 needs b12x (sm_121) kernels, which only Blackwell GPUs expose. The 3090 is Ampere SM86. Use the GGUF path, not NVFP4, and expect the 1-bit class on a 3090 + 96 GB to land between 3 and 8 tokens per second with hybrid offload — treat that as an estimate, not a benchmark.

CPU bandwidth is the bottleneck, not the GPU

Once you accept that the 24 GB VRAM card holds the active 6B and the rest of the experts streams in from system RAM, the math is straightforward: every decoded token pulls a few routed experts from CPU memory into the GPU on the fly. The ceiling is DDR bandwidth, not CUDA cores — the trips across the PCIe bus and into RAM, not the GPU itself.

DDR5 quad-channel is roughly 2x faster than DDR4 dual-channel at the same core count on this workload. A 128 GB DDR5 machine outruns a 96 GB DDR4 one even though the total capacity looks similar on paper. MoE expert dispatch is also CPU-bound, so set --threads to your physical core count (32 is a reasonable default for a modern desktop); hyperthreading does not help here.

Recipe — llama.cpp with UD-IQ1_S and hybrid GPU/CPU offload

This is the proven path for Flash-Next on consumer hardware today. Build llama.cpp with CUDA, download the Unsloth 1-bit GGUF, and split layers between the 3090 and system RAM.

# 1. Build llama.cpp with CUDA support
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first \
    --target llama-cli llama-server llama-gguf-split

# 2. Download the Unsloth 1-bit GGUF (72.5 GB)
hf download unsloth/Qwen3.8-Flash-Next-GGUF \
    --local-dir unsloth/Qwen3.8-Flash-Next-GGUF \
    --include "*UD-IQ1_S*"

# 3. Run with hybrid GPU/CPU layer split.
# --n-gpu-layers controls how many of the active experts live on the GPU.
# For MoE models, "all routed experts" offload is needed; the Unsloth
# GGUF already separates routed vs shared experts for this purpose.
./llama.cpp/llama-cli \
    --model unsloth/Qwen3.8-Flash-Next-GGUF/Qwen3.8-Flash-Next-UD-IQ1_S.gguf \
    --ctx-size 32768 \
    --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
    --n-gpu-layers 99 \
    --threads 32 \
    --seed 3407

# 4. Or serve via llama-server for an OpenAI-compatible API
./llama.cpp/llama-server \
    --model unsloth/Qwen3.8-Flash-Next-GGUF/Qwen3.8-Flash-Next-UD-IQ1_S.gguf \
    --alias "Qwen3.8-Flash-Next" \
    --ctx-size 32768 \
    --port 8001

Two flags that matter:

• --n-gpu-layers 99 is the practical max for keeping the active path on the 3090. The routed-experts tensor has to land on the GPU; if n-gpu-layers is too low the decode will constantly fault experts in from RAM. Start high and reduce only if you OOM the GPU.
• --ctx-size 32768 is the honest starting context. QSA keeps the cache tiny (about 25 KB/token) so the 3090’s VRAM goes further than you’d expect, but 262K will still push you into the danger zone. Start at 32K; raise only as needed.

The MTP caveat

The MTP head is in the checkpoint. vLLM and SGLang expose it. llama.cpp does not yet have a verified MTP path for this model. If you need the 1.7-2x speculative-decode speedup, swap to vLLM:

vllm serve unsloth/Qwen3.8-Flash-Next-GGUF \
    --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
    --max-model-len 32768 \
    --enable-prefix-caching

num_speculative_tokens=3 is the Qwen-family sweet spot. 1-2 gives no speedup; 4+ collapses acceptance rate and slows you down. Start at 3, verify on a benchmark, adjust only if your acceptance-rate curve disagrees.

Reasoning mode is on by default

Qwen3.8-Flash-Next ships with thinking on. From the model card:

“Qwen3.8-Flash-Next models operate in thinking mode by default, generating thinking content signified by \n…\n\n before producing the final responses.” — Qwen Team, HF model card, 2026-08-26.

Make your peace with <think>...</think> blocks, or pass chat_template_kwargs={"enable_thinking": False} to turn them off. For agent work, leave thinking on.

The temptation is to drop reasoning_effort (the vendor’s three-level knob that controls how long the model’s internal <think>…</think> chain runs; default xhigh) to low to claw back speed on this slow build. Do not do it for agent loops:

“In multi-turn agentic tasks, lower reasoning effort does not always reduce overall task completion time. Although it may produce faster per-turn responses, it can also lead to insufficient analysis, more failures, and repeated retries, which may increase total latency and token consumption.” — Qwen Team, HF model card, 2026-08-26.

The vendor explicitly says the shortcut backfires on agentic work. xhigh (the default) is correct for agents; use low only for short single-shot completions where you have already validated the output shape.

Why the dense 27B is still the better pick

The honest comparison on a 24 GB card, from the Atomic Chat first-party guide:

“If your machine is a 24 GB card, the dense 27B remains the stronger pick for local coding; Flash Next is the better model when you have the memory for it.” — Andrew Dyuzhov, Atomic Chat guide, 2026-08-25.

Flash-Next wins launch benchmarks, but not if your hardware runs it at 3-8 tok/s while the dense 27B at Q4_K_M posts ~80 tok/s at 125K context ([INFERENCE — extrapolation], from the prior ABS brief on the 27B) — roughly 10x faster, and quality is not 10x better.

Use the 27B for anything agentic or IDE autocomplete. Use Flash-Next only if you have the 96 GB tier and patience for 3-8 tok/s, you want the launch numbers (DeepSWE 58.7 vs 42.2; IFBench 81.3 vs 79.5), or you are benchmarking rather than shipping.

Pitfalls worth restating

• UD-IQ1_S at 72.5 GB is the realistic ceiling on a 3090 + 96 GB. UD-Q4_K_XL is 110 GB and requires RAM you do not have.
• CPU bandwidth, not GPU, is the bottleneck. DDR5 quad-channel is roughly 2x faster than DDR4 dual-channel on this build.
• Do not confuse the 4-bit GGUF (~110 GB) with the BF16 checkpoint (~360 GB). Only the GGUF is runnable on consumer hardware.
• MTP is not yet verified in llama.cpp for this model. Use vLLM or SGLang if you need the speculative speedup.
• Reasoning mode is on by default. Do not drop reasoning_effort to low for agent work.
• KV cache still dominates VRAM at long context. Start at 32K; QSA keeps the per-token cost low so you can go further than on the 27B, but not infinitely.
• Fake GGUF repos appeared on HF before the official drop. Download only from Qwen/Qwen3.8-Flash-Next, unsloth/Qwen3.8-Flash-Next-GGUF, or AtomicChat/Qwen3.8-Flash-Next-GGUF.

Done means

• llama-server (or vllm serve) is up and /v1/models returns 200.
• The active 6B is on the 3090; the rest streams from system RAM at expected 3-8 tok/s.
• Context sits at 32K and the model fits without page-thrashing.
• reasoning_effort is xhigh (default) for agent work; low only for validated single-shot completions.
• The download is from unsloth/Qwen3.8-Flash-Next-GGUF or the official Qwen mirror — not from a third-party gguf-... lookalike.

What this article does NOT cover

• Mac/Apple Silicon setup — different memory model, see the Mac guide.
• DGX Spark / GB10 NVFP4 path — different kernel class, see the DGX Spark guide.
• The hosted Qwen3.8-Flash on Qwen Cloud — that is the served version of the same weights with 1M context and built-in tools, not the local-inference target.
• The Qwen3.8-Max 2.4T flagship — API-only, not relevant to local inference.
• Training or fine-tuning — weights are inference-only.

Related guides

• Qwen3.8-27B on an RTX 3090 — the dense 27B this article keeps recommending instead
• Qwen3.8-27B on a Mac
• Qwen3.8-27B on DGX Spark
• Will the frontier-model stack fit your hardware?

Sources

• Qwen3.8-Flash-Next - Hugging Face model card
• Qwen3.8-Flash-Next - GitHub repo (tech report PDF)
• Atomic Chat - How to run Qwen3.8-Flash-Next locally
• Unsloth Dynamic 3.0 GGUF docs
• ggml-org/llama.cpp - GitHub
• Daniel Lougen on X - dual RTX PRO 6000 NVFP4 + MTP first-look numbers
• AJ on X - Unsloth UD-IQ1_S at 72.5 GB drop
• NVIDIA dev forum - Qwen3.8-Flash-Next thread
• Research brief this article was built from: coordinate/abs-research-briefs/qwen3.8-flash-next-brief.md (R2, Marvin-authored, 2026-08-26)