2026-09-27

How I got Qwen3.8 27B decoding 2.4x faster on an AMD R9700

After the 3090 work, a friend gave me remote access to an AMD Radeon AI PRO R9700 (RDNA4, gfx1201). I'd never done inference on AMD before. I wanted to know how much of what I learned tuning Qwen3.8 27B on the 3090 carries over to a different vendor and a different backend.

I spent one day on it (Sep 27, 2026), all on llama.cpp's Vulkan backend. On short prompts, decode went from 36 tok/s on stock llama.cpp to 88 to 96 tok/s with MTP speculative decoding and a few RDNA4 patches. With a combined drafter it reached 138 tok/s on JSON. This is a smaller project than the CUDA ones, and it isn't finished.

The fork is TheRealBluesun/llama.cpp, branch r9700: 10 commits on top of upstream llama.cpp as of Sep 26.

Baseline

Stock llama.cpp, Vulkan, UD-IQ4_XS quant, no speculative decoding:

None of the CUDA kernels transfer. The general approach does: speculative decoding, cheaper drafting and fewer state copies.

What I changed

RDNA4 mat-vec tuning (commit d7051a9). On RDNA3, llama.cpp's Vulkan backend already caps rows per workgroup for the int-dot mat-vec path at 4 or more columns. RDNA4 didn't have that rule. It matters for speculative verify, where the target multiplies a handful of columns at once. Per the commit notes, at 8 columns K-quants go from ~370-425 to ~570-595 GB/s. The same commit forces the q6_K MMVQ path for batches on RDNA4. MMVQ holds ~95% of bandwidth up to 8 columns, while the float path drops to ~46%. At one column they tie.

Magic bias on the RDNA4 cm1 epilogue (commit fc314e1). The cooperative-matrix int8 matmul shader converts accumulators with a float magic-bias trick, but the trick was disabled for wave32. I enabled it on RDNA4 for the symmetric quant types only.

Fused GDN snapshot copy (commit 6abed85). Most of Qwen3.8 27B is gated delta net (GDN) linear attention. During speculative decoding the GDN op writes a state snapshot per draft position, and a separate CPY node then copies those snapshots into the recurrent cache. I fused the two in the Vulkan backend, so the shader writes the snapshots directly into the recurrent cache.

MTP speculative decoding, k=3. The drafter is Qwen's own MTP head. Together with the patches above, this accounts for most of the gain in the results table below.

Reduced-vocab MTP draft head (commit e52a191). The full LM head takes about 40% of a draft step, and the drafter only needs the top candidates. An optional draft-only head covers a subset of frequent tokens. It computes logits for that subset and scatters them into a full-vocab row filled with a large negative value, so the sampler and everything after it stay the same. The target still verifies with the full head. My vLLM fork does the same thing with an FP8 hot-token draft head.

Combined drafters (commits b5f2b62 and b22fb9f). MTP and the DFlash2 drafter do well on different workloads, so I run both. The server gets a second context for MTP. DFlash2 drafts first on each request, gated by an EMA of its accepted tokens (alpha 0.3). If the EMA drops below a threshold, or the context passes a set position, DFlash2 declines and MTP drafts instead. Every N steps a probe step keeps the EMA updated so DFlash2 can come back. New flags: --spec-mtp-n-max, --spec-dflash-ctx-max, --spec-dflash-min-acc, --spec-dflash-probe. I also sized the block-draft context to one verify block instead of the full context.

The remaining commits add experiment knobs (GGML_VK_MMV_ROWS_N, GGML_VK_RDNA4_CM1_ALL), profiling for MTP process sync cost, and backend-op tests for the shapes involved.

Results

Qwen3.8 27B, stock vs tuned (tok/s):

The two rows use different quants. This table compares whole setups and doesn't isolate the patches.

Combined drafters, short prompts (greedy tok/s):

Decode after a long prompt (tok/s):

DFlash2 is much faster on structured output like JSON and code, and slows down as context grows. MTP holds up better at depth. The combined setup keeps most of each one's gains and loses a few percent at long context.

Next

• A prefill GEMM kernel for RDNA4.
• A cleaner way to pick DFlash2 or MTP per request than the EMA gate.
• Upstreaming. Each commit is tagged with a PR group, so they can go to llama.cpp as separate PRs: the RDNA4 mat-vec rule, the cm1 magic bias, the GDN snapshot fusion, the reduced-vocab draft head and the multi-drafter support.

Code

github.com/TheRealBluesun/llama.cpp, branch r9700.

Code: TheRealBluesun/llama.cpp