Qwen3.8-27B on a single RTX 3090: MTP via llama.cpp-from-source or vLLM

This is a recipe article for Qwen3.8-27B on one RTX 3090 (24 GB). No published Qwen3.8-27B benchmark on a single 3090 exists at the time of writing. The reference numbers come from Qwen3.6-27B measurements on the same hardware class — same architecture (hybrid DeltaNet + attention), same MTP engine, same Q4_K_M quant — and Qwen3.8 is expected to land in the same speed class once the community re-runs the benchmarks. The single biggest win across both recipes is MTP, and the single biggest mistake is Ollama, which doesn’t expose MTP for Qwen3.6/3.8 in its 0.30.x build.

What Qwen3.8-27B is, and why 24 GB forces a quant

Qwen3.8-27B is dense (not Mixture-of-Experts), 27B params, Apache 2.0. The Hugging Face model card lists hidden dim 5120 across 64 hybrid layers (48 Gated DeltaNet linear-attention + 16 full-attention). Trained with multi-token prediction (MTP). Native context 262,144 tokens, extendable to 1 M via YaRN.

Per the brief’s memory ladder, in BF16 the weights alone are 55–56 GB — well past a single 3090’s 24 GB. FP8 (28.5 GB) is borderline. The realistic targets are Q4_K_M GGUF or GPTQ-Int4, both around 16.5–17.1 GB. Add a Q8 KV cache and a 16K–32K context window and you land cleanly inside the 3090’s 24 GB.

Recipe 1 — llama.cpp with MTP (highest single-stream tok/s)

llama.cpp mainline doesn’t yet have MTP merged; the MTP-enabling PR is #22673. Build that branch and serve an MTP-enabled GGUF.

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git fetch origin pull/22673/head:mtp-pr
git checkout mtp-pr
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server -j

./llama-server \
  -m Qwen3.8-27B-Q4_K_M-mtp.gguf \
  --alias qwen3.8-27b \
  --host 0.0.0.0 --port 8910 \
  -ngl 99 -fa on -c 32768 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --temp 1.0 --top-k 20 --top-p 0.95 \
  --speculative --draft-max 3

Two notes. First, the KV cache dominates VRAM at long context — start at 16–32K and raise only as needed. Second, the --temp 1.0 line is intentional and matches the Qwen3.8-27B HF model card sampling table. The Qwen ReadTheDocs quickstart lists a slightly different temperature of 0.6 in one place; that’s a known doc variance. The hard rule that holds across both sources:

“For thinking mode, use Temperature=0.6, TopP=0.95, TopK=20, and MinP=0 … DO NOT use greedy decoding, as it can lead to performance degradation and endless repetitions.” — Qwen ReadTheDocs, quickstart, 2026.

Reference tok/s on this recipe: ~38 → 65 tok/s with MTP enabled, measured on Qwen3.6-27B at the same Q4_K_M on RTX 3090, per Abid Ali Awan, Datacamp MTP tutorial (behind paywall; verified manually on 2026-08-15). Qwen3.8-27B expected in the same class.

Recipe 2 — vLLM with MTP (the agents-and-tools build)

If you want OpenAI-style tool calling, structured outputs, and parallel request batching, vLLM is the better fit. The MTP integration lives behind a single flag:

vllm serve Qwen/Qwen3.8-27B-GPTQ-Int4 \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --gpu-memory-utilization 0.9

Tensor parallelism is 1 because there’s only one GPU. --reasoning-parser qwen3 strips <think>...</think> before they hit your tool-call layer. --enable-prefix-caching amortises the prefill cost across turns so multi-turn agents don’t pay it twice.

num_speculative_tokens is not “more is better.” Per NVIDIA dev forum benchmarks on a similar Qwen3.6-27B FP8 build: num_speculative_tokens=3 hits 1.94× over baseline, and tokens 2 and 4 are both slower than 3. Use 3.

Reference tok/s on this recipe: ~50–55 tok/s, measured on Qwen3.6-27B on the same vLLM+MTP-n=3 stack, per the same NVIDIA thread. Lower than raw llama.cpp because vLLM pays a per-request overhead for batching and tool parsing. Qwen3.8-27B expected in the same class.

Reasoning effort and sampling

Qwen3.8 ships with thinking on by default. Reasoning effort has three levels — xhigh (default), medium, and low — per the LM Studio Qwen3.8-27B model page:

“Thinking is enabled by default, with xhigh, medium, and low reasoning-effort levels.” — LM Studio, model page, Aug 2026.

The temptation is to crank everything to low for speed. Resist it. From the model card:

“In multi-turn agentic tasks, lower reasoning effort does not always reduce overall task completion time. Although it may produce faster per-turn responses, it can also lead to insufficient analysis, more failures, and repeated retries, which may increase total latency and token consumption.” — Qwen Team, HF model card, Aug 2026.

For short replies — single-shot classification, formatting, lightweight autocomplete — low is fine. For anything agentic that loops on its own output, keep medium (or xhigh for hard reasoning). Don’t blanket-set low.

Why this beats the stock setup

• Standard Ollama 0.30.x wraps llama.cpp but does not expose MTP for Qwen3.6 or Qwen3.8. To get the ~1.7–2× MTP speedup you must switch to llama.cpp-from-source (the MTP-enabling branch is PR #22673) or vLLM (where MTP is a --speculative-config flag).
• num_speculative_tokens=3 is the sweet spot; 2 and 4 are both slower. Verified on Qwen3.6-27B FP8 on the NVIDIA dev forum thread above.
• KV cache dominates VRAM at long context — start at 16–32K and grow only as needed.

Pitfalls worth restating

• Ollama won’t give you MTP — switch runtimes to capture the 1.7–2× speedup.
• num_speculative_tokens=3 is the sweet spot; 2 and 4 are both slower on Qwen3.6-27B.
• KV cache dominates VRAM at long context — start at 16–32K.
• AWQ-Int4 quality: the brief claimed “92% of f16 quality on hard reasoning” sourced to a paywalled Reddit overnight-stack benchmark. Unsloth’s citable vendor number (NVFP4-vs-BF16 at 92–97% top-1% accuracy) is from a different quant and isn’t a direct substitute. Treat the 92% figure as a working rule of thumb until an AWQ-vs-BF16 head-to-head ships.

Done means

• vllm serve (or llama-server) is up and returns 200 on /v1/models.
• --speculative-config (vLLM) or --speculative --draft-max 3 (llama.cpp) is active — MTP is on.
• Context stays at 16–32K and the model fits in 24 GB with KV cache to spare.
• reasoning_effort is medium (or your chosen level) and thinking blocks render.
• Throughput is in the Qwen3.6-27B reference class for the same recipe (~65 tok/s via llama.cpp+MTP, ~55 tok/s via vLLM+MTP, ~85 tok/s via overnight-stack MTP n=3 + 125K context). No Qwen3.8-27B target on a single 3090 is published yet; you are running on the same chip + stack as the Qwen3.6-27B reference.

What this article does NOT cover

• Mac/Apple Silicon setup — see the Mac guide.
• DGX Spark / GB10 specifics — see the DGX Spark guide.
• Training or fine-tuning — weights are inference-only.
• The Qwen3.8-2.4T-A95B variant — different architecture, far larger memory footprint.

Sources

• Qwen3.8-27B - Hugging Face model card
• Qwen ReadTheDocs - quickstart
• LM Studio - Qwen3.8-27B model page
• ggml-org/llama.cpp - GitHub
• ggml-org/llama.cpp PR #22673
• Abid Ali Awan, Datacamp - Multi-token prediction with llama.cpp (behind paywall; verified manually on 2026-08-15)
• NVIDIA dev forum - speculative decoding on Qwen3.6-27B-FP8 (Qwen3.6 source for the n=3 sweet-spot)
• Research brief this article was built from: coordinate/abs-research-briefs/qwen3.8-27b-hardware-tuning-brief.md (R2, Marvin-authored, 2026-08-14)