GLM-5.3-Flash on an RTX 3090: KTransformers offload, honestly slow — or wait for GGUFs

One-line job: understand exactly what running Zhipu AI’s 320B-flagship-flash on a single 24 GB card costs you today, decide whether the experiment is worth your weekend, and know which faster path you’re waiting for instead. Audience: RTX 3090 owners with large system RAM who run local models and saw the launch numbers. Not for: Mac or DGX Spark readers; anyone wanting usable interactive speed from this model class this week. Last verified: 2026-08-26

The problem

GLM-5.3-Flash is the new open-weight 320B-total / 18B-active MoE from Z.ai — hybrid attention, native multi-token prediction head, strong coding benchmarks, and weights shipping as FP8 at roughly 306 GiB. Your card holds 24 GB VRAM, and even with 96–128 GB of system RAM beside it, nothing about those two sentences adds up to local inference yet.

So before any command: the honest answer is that the only documented local path for this model is KTransformers heterogeneous CPU-expert offload — and its validated GPU list does not include your card.

What KTransformers officially supports

From the GLM-5.3-Flash tutorial (kvcache-ai, 2026-08-26):

“The current implementation supports: NVIDIA SM89 and SM120 GPUs (RTX 40 and 50 series); The AVX-512 FP8 CPU expert kernel… A context window of up to 1M tokens; Multimodality: text, multiple images, video.”

SM89 = Ada (RTX 40-series). SM120 = Blackwell consumer (RTX 50-series). Your 3090 is SM86 — Ampere, one generation below the validated floor. The tutorial documents nothing about Ampere; anything you get working there is self-supported territory: expect fallback kernels, longer warm-ups, possibly outright failure on the FP8 expert paths.

Three more rows from their own documentation define the scale of the ask:

“GLM-5.3-Flash has approximately 321B parameters. Its 45 layers comprise 34 Linear Attention layers and 11 DSA layers.”

“The FP8 model occupies approximately 306 GiB. Reserve at least 350 GB of available system memory.”

“When Layerwise Prefill is enabled… normalizes the resident GPU expert count to zero.” — meaning the --kt-num-gpu-experts figure you set stops guaranteeing residency during prefill.

The memory ladder you’re actually facing

Mirror of the Qwen Flash-Next situation on this hardware, scaled up 2.4×:

Quant tier	Estimated size	Fits 96 GB RAM?	Status
FP8 checkpoint	~306 GiB	No — needs 350 GB+ class	Official format; KTransformers’ target
~3-bit GGUF (projected)	~130 GB	No	Nothing published
~2-bit GGUF (projected)	~85-95 GiB	Yes, tight	The realistic future target
~1-bit GGUF (projected)	~50 GB	Yes, comfortably	Quality collapse likely

Nothing below the first row exists today — the rows are arithmetic on published parameter counts, not announcements. When Unsloth-style builds land, this table is where your decision lives. History says watch for day-zero drops within two weeks of launch for high-profile models.

Why your system RAM matters more than your GPU

Offloaded MoE inference is a memory-bandwidth problem wearing a GPU costume. Per token, active expert weights stream from system RAM across PCIe; VRAM holds attention states, shared layers, and whatever fits of the hot path. Consequences:

• Dual-channel DDR4 (~50 GB/s) is the bottleneck nobody can fix in software; quad-channel DDR5 roughly doubles it.
• PCIe generation matters less than RAM channels because the same weights re-stream every token rather than moving once.
• Physical cores help only until bandwidth saturates — past that, more --cpu-infer threads add scheduling overhead, not speed.

The same physics caps Qwen’s Flash-Next at 3–8 tok/s here; GLM-5.3-Flash moves more bytes per token, so expect equal-or-worse [INFERENCE — extrapolation].

If you attempt it anyway

The mechanics, as documented for validated cards — treat every line here as untested-on-SM86:

# 1. System prerequisites: ≥350 GB free RAM+swap headroom, AVX-512 CPU
#    (dual-channel DDR4 will bottleneck hard; bandwidth-bound workload)

# 2. KTransformers install with GLM support (follow repo build docs for your CUDA version)
git clone https://github.com/kvcache-ai/ktransformers.git
cd ktransformers && pip install -r requirements.txt

# 3. Weights
huggingface-cli download zai-org/GLM-5.3-Flash --local-dir ./GLM-5.3-Flash
# ~306 GiB download. Plan disk AND patience accordingly.

# 4. Launch pattern per tutorial (flags shown are the documented multi-GPU form;
#    a single-card run means nearly everything lives in system RAM anyway)
python ktransformers/local_chat.py \
  --model-path ./GLM-5.3-Flash \
  --cpu-infer <physical_cores> \
  --max-new-tokens 1024

Expected speed, flagged as estimate

No published single-3090 number exists for this model. Nearest anchors: GLM-5.2 — smaller and lighter — managed roughly 8 tok/s in a 1-bit quant across two Spark-class nodes; this model is 2–4× larger per token moved through similar bandwidth-class paths. A realistic single-3090 expectation with everything offloaded lands in single-digit tok/s [INFERENCE — extrapolation], likely dominated by memory bandwidth rather than either processor. If you test it, publish your numbers — the community has none.

What to measure if you do run it

One weekend experiment worth doing properly:

1. tok/s decode over ≥200 generated tokens, after warm-up — first-token numbers are prefill-dominated lies.
2. RAM bandwidth saturation (perf stat -e uncore_imc_0,event or your platform’s equivalent) so you can attribute slowness correctly.
3. Page-thrash check: peak RSS against installed RAM; swapping invalidates every other measurement.
4. First-token latency separately from decode speed — KDA state initialization may dominate cold starts.

Why waiting may be the smarter play

Two pipelines typically follow big open-weight launches, and both change this picture:

1. Unsloth-style low-bit GGUFs. A 2-bit build (~80 GiB) fits in 96 GB system RAM with room for context, exactly like Qwen3.8-Flash-Next’s UD-IQ1_S economics on this same hardware class. Watch Unsloth’s Dynamic quants page and the LocalLLM community channels.
2. llama.cpp architectural support. Hybrid MLA+DSA+KDA attention needs engine-level code — weeks, not days, historically. Once merged plus GGUF, llama-server becomes the one-command path KTransformers can’t be.

Your 3090 isn’t idle while you wait: Qwen3.8-Flash-Next runs today at UD-IQ1_S with an honest 3–8 tok/s and day-one community quants, and the dense 27B remains genuinely fast on this exact card.

Reasoning mode applies here too

However you eventually serve it locally, client-side behavior follows the vendor contract:

“GLM-5.3 supports three thinking effort levels: low, high, and max. Disabling thinking is no longer supported by GLM-5.3.” — Z.ai blog, 2026-08-14

If your tooling sends thinking.type: disabled, fix the payload (enabled + reasoning_effort: low) before your first request, not after debugging it.

Pitfalls worth flagging

• Treating KTransformers’ validated-GPU list as a suggestion. It’s a statement of what they tested; SM86 is off the map, full stop.
• Starting the 306 GiB download without checking disk. Reserve ≥350 GB free; failed downloads near-full disks waste hours.
• Blaming MTP absence for slowness that is memory bandwidth. Offloaded MoE inference is bandwidth-shaped; speculative decoding cannot fix physics.
• Comparing your tok/s against SGLang cluster numbers. 4×GB300 aggregates are a different universe; keep them out of single-card conversations.
• Assuming vision input works through every serving layer. Multimodal support is listed for KTransformers, but confirm the chat template path you use actually routes images.
• Confusing GLM-5.2 low-bit recipes with this model. Different attention internals mean old quant/rpc tricks don’t transfer.

Done means

• You can explain why SM89/SM120 excludes your SM86 card and what “self-supported” means for you.
• Any weight-download plan you make reserves 350+ GB and starts from the official zai-org/GLM-5.3-Flash repo only (mirror fakes appear around launches).
• You know the realistic speed band for any working offload setup is single-digit tok/s [INFERENCE — extrapolation] until real measurements exist.
• You have Unsloth and llama.cpp release channels watched for the sub-128-bit + engine-support combo that changes this answer.
• Meanwhile, you point actual work at models that fit — dense 27B locally, or hosted GLM-5.3-Flash via API when you need this capability level today.

What this article does NOT cover

• Apple Silicon paths — see the Mac sibling guide (short answer: none native yet).
• Blackwell NVFP4-class serving — see the DGX Spark sibling.
• KTransformers installation troubleshooting beyond what the official tutorial covers.
• Fine-tuning; these weights are inference-only scope here.

Related guides

• Qwen3.8-Flash-Next on an RTX 3090 — the frontier-MoE with a working (if honest) day-zero path on this exact card
• Will the frontier-model stack fit your hardware?
• vLLM setup on a single GPU — when your goal is serving models that DO fit

Sources

• KTransformers GLM-5.3-Flash tutorial
• Z.ai model card on Hugging Face
• SGLang cookbook — verified matrix (contrast)
• Z.ai GLM-5.3 launch blog