Qwen3.8 Flash-Next on Apple Silicon: 4-bit Needs About 110GB of a 128GB Mac

What shipped

Alibaba’s Qwen team released Qwen3.8-Flash-Next on 2026-08-26. First open-weight model on the Qwen4 architecture: 125B/6B MoE with native MTP, hybrid Gated DeltaNet (GDN) plus Qwen Sparse Attention (QSA), and a 51B N-gram embedding table on top. About 180B parameters on disk. Native context: 262,144 tokens, extensible to 1,000,000 via YaRN.

The vendor card opens with the headline: “Qwen3.8-Flash-Next features a 125B-parameter main model, supplemented by an additional 51B N-gram embeddings, with 6B parameters activated per token. Compared with Qwen3.7-Plus, Qwen3.8-Flash-Next substantially reduces both training and inference cost — training takes only about 1/9 as much, yet it delivers superior capabilities in coding and office tasks.” (Qwen GitHub repo, 2026-08-26).

For a 128 GB Mac Studio or MacBook Pro with M-series Pro/Max/Ultra silicon, this is runnable — and GGUF-on-Metal (a GGUF — GPT-Generated Unified Format, the .gguf file format used by llama.cpp — Georgi Gerganov’s C++ inference engine, the default local runtime for GGUF models — with Metal — Apple’s GPU API used by llama.cpp’s --metal flag for Apple Silicon) is the only path that works today. The recipe is one macOS sysctl (iogpu.wired_limit_mb — the macOS sysctl that caps the Metal driver at ~75% of unified memory by default) plus the day-zero Unsloth + AtomicChat GGUF at the UD-Q4_K_XL (UD-Q4_K_XL — Unsloth Dynamic 4-bit K-quant extra-large, a 4-bit quantization with mixed-precision sub-weights) quant. The architectural lever is QSA’s tiny KV cache: ~25 KB per token, so a full 262K-token context only adds ~6.4 GB of KV state on top of the ~110 GB GGUF. That is the path this article walks.

Why a Mac can run this at all

The headline architectural lever is the N-gram embedding table. From the model card: “Embeddings provide a unique axis for parameter scaling that requires less computation and is more amenable to offloading than Mixture-of-Experts (MoE). By indexing with short n-grams, this approach makes parameter scaling highly efficient for memory-constrained accelerators without sacrificing quality.” (Hugging Face model card, 2026-08-26). On Apple Silicon the 51B table lives in unified memory and is prefetched asynchronously — readers do not need 180 GB of VRAM. A 96 GB Mac serves the model; a 128 GB Mac runs the 4-bit class at full 262K context with headroom.

The second lever is QSA’s tiny KV cache. Andrew Dyuzhov’s Atomic Chat guide: “The Qwen4 architecture keeps [KV cache] cost small. Of the 48 layers, 36 use Gated DeltaNet with a fixed-size state that does not grow with the conversation, and the 12 Qwen Sparse Attention layers keep just 2 KV heads each. The attention cache works out to about 25 KB per token, a tenth of what the dense 27B stores.” (Atomic Chat guide, 2026-08-25). At ~25 KB/token, a full 262K-token context holds about 6.4 GB of KV state — small enough that the GGUF, the table, and the cache coexist in 128 GB without paging.

Memory ladder on Mac

Memory ladder (Unsloth Dynamic 3.0 + AtomicChat):

Mac unified memory	Recommended quant	Approx. size	Notes
64 GB	—	—	No safe build
80 GB	1-bit class	~73 GB	Short context only
96 GB	2-bit class	~88 GB	Full native context, tight headroom
128 GB	UD-Q4_K_XL	~110 GB	Full 262K context with room to spare
192 GB or more	6-bit class	~165 GB	Near-reference quality

The 128 GB tier is the practical sweet spot. Below 96 GB, this model is not the right tool — pick the dense Qwen3.8-27B instead (see the Qwen3.8-27B Mac setup guide for the matching dense recipe).

Recipe: 128 GB Mac, GGUF + llama.cpp

This is the day-zero working path. All commands are copy-pasteable; the only sudo is the one-line sysctl that raises the Metal memory cap.

# 1. Build llama.cpp with Metal support.
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_METAL=ON
cmake --build llama.cpp/build --config Release -j --clean-first \
    --target llama-cli llama-mtmd-cli llama-server llama-gguf-split

# 2. Pull the Unsloth Dynamic GGUF (4-bit class, ~110 GB).
#    AtomicChat also publishes a GGUF at the same path scheme:
#       AtomicChat/Qwen3.8-Flash-Next-GGUF
#    Both are first-party — never link random third-party "gguf-qwen3.8-flash-next" mirrors.
pip install -U "huggingface_hub"
hf download unsloth/Qwen3.8-Flash-Next-GGUF \
    --local-dir unsloth/Qwen3.8-Flash-Next-GGUF \
    --include "*UD-Q4_K_XL*" \
    --include "*mmproj-F16*"     # omit if you do not want vision

# 3. IMPORTANT: raise the Metal memory cap BEFORE running.
#    macOS caps Metal at ~75% of unified memory by default.
#    For 128 GB, raise to at least 110 GB. Adjust the number for other tiers:
#      96 GB  -> 80000  (80 GB)
#     192 GB -> 170000 (170 GB)
sudo sysctl iogpu.wired_limit_mb=110000

# 4. Run llama-cli with the vendor sampling defaults (thinking mode default).
./llama.cpp/llama-cli \
    --model unsloth/Qwen3.8-Flash-Next-GGUF/Qwen3.8-Flash-Next-UD-Q4_K_XL.gguf \
    --mmproj unsloth/Qwen3.8-Flash-Next-GGUF/mmproj-F16.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00 \
    --ctx-size 262144

# 5. Or serve via llama-server for an OpenAI-compatible API.
./llama.cpp/llama-server \
    --model unsloth/Qwen3.8-Flash-Next-GGUF/Qwen3.8-Flash-Next-UD-Q4_K_XL.gguf \
    --alias "Qwen3.8-Flash-Next" \
    --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
    --ctx-size 262144 \
    --port 8001

The iogpu.wired_limit_mb line is the foot-gun. Without it, macOS refuses to wire the full ~110 GB to Metal and the GGUF fails to load at full context. The setting survives until reboot; persist it via /etc/sysctl.conf if you want it permanent.

Thinking mode and the off-switch

The vendor defaults to thinking mode. From the card: “Qwen3.8-Flash-Next models operate in thinking mode by default, generating thinking content signified by <think>\n...</think>\n\n before producing the final responses.” (Hugging Face model card, 2026-08-26). For direct answers, pass chat_template_kwargs={"enable_thinking": False} to the server or chat client — otherwise every response is preceded by a <think> block you may not want.

reasoning_effort levels are xhigh (default), medium, and low. Do not drop reasoning effort for agent or coding work. The vendor warning is explicit: “In multi-turn agentic tasks, lower reasoning effort does not always reduce overall task completion time. Although it may produce faster per-turn responses, it can also lead to insufficient analysis, more failures, and repeated retries, which may increase total latency and token consumption.” (Hugging Face model card, 2026-08-26). xhigh is the default; leave it there unless you have a measured reason to lower it.

Sampling defaults (copy verbatim)

The vendor’s published sampling recipes differ between thinking and instruct modes:

Param	Thinking mode	Instruct / non-thinking mode
temperature	1.0	0.7
top_p	0.95	0.80
top_k	20	20
min_p	0.0	0.0
presence_penalty	0.0	1.5
repetition_penalty	1.0	1.0

The temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00 flags in the recipe match thinking mode; pass chat_template_kwargs={"enable_thinking": False} with --temp 0.7 --top-p 0.80 for direct, non-thinking answers.

What the Mac path does not yet give you

A few honest gaps:

• No published Mac tok/s number yet. Closest published figure is dual RTX PRO 6000 (NVFP4, with MTP) at 212–259 tok/s — that’s 96 GB/GPU Blackwell with b12x kernels. Do not extrapolate it to a Mac. A 128 GB Mac running UD-Q4_K_XL on Metal should land in the same class as the dense 27B’s measured Mac numbers (single-digit to low-double-digit tok/s decode, depending on context), but no third-party benchmark for Flash-Next on Mac exists yet.
• MLX-native builds may lag. If a faster MLX build lands, it will probably go through the same recipe with an MLX-aware runner.
• MTP in llama.cpp is not yet verified for this model. The 1-layer MTP head is in the checkpoint; vLLM and SGLang support it (e.g. --speculative-config '{"method":"mtp", "num_speculative_tokens":3}'); llama.cpp may or may not. Do not promise MTP speedup on Mac today.
• Vision is optional but adds ~1 GB. Skip --mmproj unsloth/Qwen3.8-Flash-Next-GGUF/mmproj-F16.gguf if you only want text.

Pitfalls worth flagging

1. Raise iogpu.wired_limit_mb first. macOS caps Metal at ~75% of unified memory by default. The 4-bit class at ~110 GB will not load otherwise.
2. Don’t conflate this with Qwen3.8-27B or Qwen3.8-Max. Three SKUs: 27B dense, Max 2.4T API-only, Flash-Next open MoE. On a 64 GB Mac, the dense 27B is the better pick.
3. Don’t drop reasoning_effort to low for agent work. Vendor warns it causes more failures and longer total time despite faster per-turn responses.
4. Don’t extrapolate the 212–259 tok/s dual-RTX-PRO-6000 number to a Mac. That is 96 GB/GPU Blackwell + NVFP4 + MTP n=3. A Mac with UD-Q4_K_XL is a different memory bandwidth tier.
5. Watch for fake GGUF/MLX repos. Fake mirrors appeared on Hugging Face before the official drop. Pull only from Qwen/Qwen3.8-Flash-Next, unsloth/Qwen3.8-Flash-Next-GGUF, or AtomicChat/Qwen3.8-Flash-Next-GGUF. Walk away from any gguf-qwen3.8-flash-next path from an unknown owner.
6. At 96 GB, expect quality drop at the bottom of the ladder. The 2-bit class at ~88 GB fits full native context but tightly; vendor guidance: “take the larger of two neighboring files” if you can.
7. MTP head is in the checkpoint, not yet a llama.cpp speedup. The recipe above does not turn it on; verify before claiming any MTP speedup on Mac.
8. Leave headroom for macOS. Browsers, Mail, Docker each pull 4–8 GB; budget ~8 GB for the OS or you swap-thrash.

Done means

• llama-server is up on port 8001 with --ctx-size 262144 and the chosen quant loaded.
• A chat request returns tokens; thinking-mode outputs are wrapped in <think>...</think>.
• reasoning_effort is at the default xhigh unless you have a measured reason to change it; pass chat_template_kwargs={"enable_thinking": False} when you want direct answers.
• iogpu.wired_limit_mb was raised before launch and is still in effect (sysctl iogpu.wired_limit_mb reports the new value).
• The chosen UD-Q4_K_XL fits in unified memory with at least 8 GB headroom left for macOS.

What this article does NOT cover

• Linux/Windows installs — need GGUF + llama.cpp on CUDA or vLLM (DGX Spark and RTX 3090 guides cover those).
• Fine-tuning Qwen3.8-Flash-Next — inference-only setups.
• Multi-GPU or multi-node clusters — single Mac only.
• The hosted Qwen3.8-Flash on Qwen Cloud — same weights, API-only with 1M context and built-in tools; that is a different article.

Related guides

Sibling hardware recipes for the dense Qwen3.8-27B and for cross-tier runs:

• Qwen3.8-27B on a Mac (M-series) — the dense-27B day-zero path — the right pick for a 64–96 GB Mac.
• Qwen3.8-27B on an NVIDIA DGX Spark — dense-27B on Blackwell-class unified memory.
• Qwen3.8-27B on an RTX 3090 (24 GB) — dense-27B on consumer NVIDIA hardware.

Broader AI-strategy and local-runtime context:

• Frontier-model stack fit — picking the right model for your hardware tier.
• The AI delivery gap — why model quality alone doesn’t ship products.
• The AI engineering skills map — what you actually need to build with models.
• Local LLMs with Ollama — the easy mode — a higher-level runner over llama.cpp if you don’t need GGUF-on-Metal control.
• Local LLMs with vLLM on a single GPU — the CUDA-side counterpart when you’re not on Apple Silicon.

Sources

• Qwen3.8-Flash-Next — Hugging Face model card
• Qwen3.8-Flash-Next launch blog — Qwen team
• Qwen3.8-Flash-Next GitHub repo (with tech_report.pdf)
• Unsloth Dynamic 3.0 GGUF documentation
• Atomic Chat — How to run Qwen3.8-Flash-Next locally
• llama.cpp GitHub repository
• Research brief this article was built from: /opt/data/a2a-work/t_20260826T225618Z_2733/qwen3.8-flash-next-brief.md (Marvin-authored, 2026-08-26)