LLMs, VLAs and speech-to-text on NPU¶
End-to-end inference of decoder-only LLMs, with text or image + text input, vision-language-action (VLA) policies and speech-to-text models on the AMD NPU2 via MLIR-AIR, with bf16 and 4-bit (Q4NX, Q4_0) weights. Models with an FLM link reimplement AMD NPU LLM designs originally developed by the FastFlowLM team, using the higher-level abstractions of the MLIR-AIR dialect; the link points to the original FastFlowLM code. Coverage and performance below are refreshed nightly. Per-model details and source are in programming_examples/llms/.
Nightly LLM Benchmark (NPU2)¶
End-to-end LLM inference performance on NPU2. Each row names the hardware it was measured on: the benchmark runner's rows are refreshed nightly, rows from other machines are measured offline. TTFT is time to first token (prefill latency); Decode is steady-state generation throughput.
Hardware
- Krackan Point: AMD Ryzen AI 5 PRO 340 / Ryzen AI 7 PRO 350, NPU2
- Strix Halo: AMD Ryzen AI Max+ 395, NPU2
Verify (make verify): pass fail skipped. — not measured.
Decode throughput vs context (tok/s)¶
Steady-state decode throughput at increasing KV-cache depth.
| Model | Hardware | Weights | 1k | 2k | 4k | 8k | 32k | 64k | 128k | KLD | GSM8K | MMLU | Verify |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| gemma-3-4b-it (FLM, HF) | Krackan Point | Q4NX | — | 21.09 | — | 15.59 | 7.52 | — | 2.40 | 0.0925 | 71 / 80 | 62 / 63 | |
| gemma-4-E2B-it (FLM, HF) | Krackan Point | Q4NX | — | 36.07 | — | 33.73 | 27.07 | — | 15.21 | 66 / 68 | 61 / 61 | ||
| gemma-4-E4B-it (FLM, HF) | Krackan Point | Q4NX | — | 17.27 | — | 15.62 | 11.20 | — | 5.22 | 80 / 78 | 72 / 72 | ||
| gpt-oss-20b (FLM, HF) | Strix Halo | Q4NX, MXFP4 experts | 17.62 | — | 15.82 | 13.92 | 8.12 | 5.20 | — | ||||
| Granite-3.3-2B (HF) | Krackan Point | Affine Q4 | — | 31.85 | — | 19.81 | 7.93 | — | 2.33 | 0.0397 | 61 / 55 | 55 / 58 | |
| LFM2-1.2B (FLM, HF) | Krackan Point | Q4_0 | — | 65.02 | — | 47.86 | 23.38 | — | 7.68 | 0.0011 | 53 / 67 | 55 / 56 | |
| Llama-3.1-8B-Instruct (FLM, HF) | Krackan Point | Q4NX | — | 13.39 | — | 11.02 | 6.38 | — | 2.34 | 0.0137 | 80 / 81 | 68 / 69 | |
| Llama-3.2-1B (HF) | Krackan Point | bf16 | — | 9.03 | — | 4.43 | 1.16 | — | 0.23 | 0.0056 | 39 / 41 | 48 / 49 | |
| Llama-3.2-1B-Instruct (FLM, HF) | Krackan Point | Q4NX | — | 66.68 | — | 48.64 | 23.48 | — | 7.68 | 0.0107 | 37 / 41 | 48 / 49 | |
| Llama-3.2-3B (HF) | Krackan Point | bf16 | — | 3.54 | — | 0.86 | 0.27 | — | — | 0.0031 | 75 / 76 | 65 / 65 | |
| Llama-3.2-3B-Instruct (FLM, HF) | Krackan Point | Q4NX | — | 26.73 | — | 17.69 | 7.58 | — | 2.31 | 0.0109 | 72 / 77 | 64 / 65 | |
| Mistral-7B-Instruct-v0.3 (HF) | Strix Halo | Affine Q4 | — | 11.17 | — | 8.88 | 4.87 | — | — | 0.0115 | 46 / 51 | 58 / 61 | |
| OLMo2-1B (HF) | Krackan Point | Affine Q4 | — | 26.28 | — | — | — | — | — | 0.0152 | 56 / 63 | 42 / 42 | |
| OpenELM-1.1B (HF) | Krackan Point | Affine Q4 | — | 34.80 | — | — | — | — | — | ||||
| Phi-2 (HF) | Krackan Point | Affine Q4 | — | 6.80 | — | — | — | — | — | 0.0246 | 56 / 60 | 61 / 61 | |
| Phi-4-mini-instruct (FLM, HF) | Krackan Point | Q4NX | — | 22.43 | — | 15.08 | 6.56 | — | 2.01 | 0.0274 | 81 / 81 | 68 / 68 | |
| Qwen2.5-0.5B (HF) | Krackan Point | bf16 | — | 10.27 | — | 5.82 | 1.62 | — | 0.29 | 0.0030 | 35 / 41 | 49 / 48 | |
| Qwen2.5-1.5B (HF) | Krackan Point | bf16 | — | 6.31 | — | 3.59 | 0.82 | — | 0.17 | 0.0019 | 69 / 62 | 63 / 63 | |
| Qwen2.5-3B (HF) | Krackan Point | bf16 | — | 3.44 | — | 1.95 | 0.48 | — | 0.10 | 0.0108 | 68 / 56 | 67 / 69 | |
| Qwen2.5-3B-Instruct (FLM, HF) | Krackan Point | Q4_0 | — | 21.23 | — | 15.57 | 7.56 | — | 2.49 | 0.0262 | 59 / 56 | 69 / 69 | |
| Qwen2.5-7B-Instruct (FLM, HF) | Krackan Point | Q4NX | — | 12.81 | — | 10.21 | 5.64 | — | 2.02 | 0.0183 | 69 / 76 | 73 / 74 | |
| Qwen2.5-7B-Instruct (FLM, HF) | Strix Halo | Q4NX | — | 11.02 | — | 8.66 | 4.69 | — | 1.55 | 0.0183 | 69 / 76 | 73 / 74 | |
| Qwen3.5-0.8B (FLM, HF) | Krackan Point | Q4NX | — | 51.38 | — | 45.44 | 30.40 | — | — | 30 / 54 | 49 / 50 | ||
| Qwen3-0.6B (HF) | Krackan Point | bf16 | — | 6.33 | — | 2.64 | 0.51 | — | — | 0.0121 | 42 / 44 | 44 / 42 | |
| Qwen3-1.7B (HF) | Krackan Point | bf16 | — | 4.80 | — | 2.58 | 0.48 | — | — | 0.0120 | 63 / 67 | 59 / 59 | |
| Qwen3-4B (HF) | Krackan Point | bf16 | — | 2.35 | — | 1.25 | 0.25 | — | — | 0.0043 | 78 / 75 | 72 / 73 | |
| Qwen3-4B (FLM, HF) | Krackan Point | Q4NX | — | 19.74 | — | 12.99 | 5.56 | — | 1.69 | 0.0306 | 80 / 75 | 72 / 73 | |
| Qwen3-8B (FLM, HF) | Krackan Point | Q4NX | — | 12.06 | — | 9.17 | 4.70 | — | 1.59 | 0.0189 | 87 / 86 | 77 / 76 | |
| Qwen3-8B (FLM, HF) | Strix Halo | Q4NX | — | 10.16 | — | 8.02 | 4.38 | — | 1.55 | 0.0189 | 87 / 86 | 77 / 76 | |
| SmolLM2-1.7B (HF) | Krackan Point | bf16 | — | 4.75 | — | 2.18 | — | — | — | 0.0061 | 42 / 42 | 51 / 51 | |
| SmolLM3-3B (HF) | Krackan Point | Affine Q4 | — | 26.00 | — | 16.94 | 7.11 | — | — | 0.0236 | 70 / 71 | 60 / 63 |
— not swept at this context, or an expected failure.
Accuracy (the decode table's KLD, GSM8K and MMLU; the prefill table's KLD).
The NPU is compared with the same model on the CPU (PyTorch): on wikitext-2 the
way llama.cpp's llama-perplexity --kl-divergence does, and on benchmark
questions. Measured offline; the NPU gives the same accuracy on every NPU2
machine. Hover over a number for its details.
- KLD: KL divergence of the NPU's next-token distribution from the CPU's with the same weights (4-bit weights dequantized on the CPU); 0 means identical. It measures the NPU implementation alone.
- GSM8K / MMLU: % correct, NPU / original bf16 checkpoint on the CPU, on the same questions (GSM8K 100, MMLU 570). For 4-bit models this includes the cost of the 4-bit weights.
Prefill latency vs prefill length (TTFT, ms)¶
Time to first token at increasing prefill length. A model with a fused prefill is timed on a prompt of exactly that many tokens, one build serving every length. A per-op prefill pads the prompt to the length its ELFs were built for, so each of its columns is a separate build and TTFT tracks that length rather than the number of real prompt tokens.
| Model | Hardware | Weights | 128 | 512 | 1k | 2k | 4k | 8k | 32k | KLD | Verify |
|---|---|---|---|---|---|---|---|---|---|---|---|
| d3-edge (FLM, HF) | Krackan Point | bf16 | — | 80 | 153 | 318 | 718 | — | — | ||
| d3-lite (FLM, HF) | Krackan Point | Q4NX | — | 149 | 287 | 591 | 1258 | — | — | ||
| gemma-3-4b-it (FLM, HF) | Krackan Point | Q4NX | — | 319 | 582 | 1159 | 2478 | 5241 | 25787 | 0.0255 | |
| gemma-4-E2B-it (FLM, HF) | Krackan Point | Q4NX | — | 298 | 566 | 1212 | 2634 | 6077 | 42984 | ||
| gemma-4-E4B-it (FLM, HF) | Krackan Point | Q4NX | — | 479 | 886 | 1880 | 4027 | 8873 | 54374 | ||
| gpt-oss-20b (FLM, HF) | Strix Halo | Q4NX, MXFP4 experts | 726 | 980 | — | 2535 | — | — | — | ||
| Granite-3.3-2B (HF) | Krackan Point | Affine Q4 | — | 286 | 569 | 1437 | 3891 | — | — | 0.0048 | |
| LFM2-1.2B (FLM, HF) | Krackan Point | Q4_0 | — | 134 | 244 | 538 | 1246 | 3049 | 25988 | 0.0027 | |
| Llama-3.1-8B-Instruct (FLM, HF) | Krackan Point | Q4NX | — | 582 | 1170 | 2592 | 6290 | 17172 | 178040 | 0.0021 | |
| Llama-3.2-1B (HF) | Krackan Point | bf16 | — | 253 | 457 | 886 | 1784 | — | — | 0.0052 | |
| Llama-3.2-1B-Instruct (FLM, HF) | Krackan Point | Q4NX | — | 121 | 236 | 582 | 1575 | 4678 | 55250 | 0.0035 | |
| Llama-3.2-3B (HF) | Krackan Point | bf16 | — | 646 | 1120 | 2197 | 5310 | — | — | 0.0054 | |
| Llama-3.2-3B-Instruct (FLM, HF) | Krackan Point | Q4NX | — | 278 | 554 | 1274 | 3350 | 9759 | 110703 | 0.0024 | |
| Mistral-7B-Instruct-v0.3 (HF) | Strix Halo | Affine Q4 | — | 1442 | 2746 | 5436 | 16494 | — | — | 0.0055 | |
| OLMo2-1B (HF) | Krackan Point | Affine Q4 | — | 161 | 286 | 614 | — | — | — | 0.0038 | |
| OpenELM-1.1B (HF) | Krackan Point | Affine Q4 | — | 196 | 397 | 987 | — | — | — | ||
| Phi-2 (HF) | Krackan Point | Affine Q4 | — | 358 | 713 | 1689 | — | — | — | 0.0257 | |
| Phi-4-mini-instruct (FLM, HF) | Krackan Point | Q4NX | — | 324 | 638 | 1463 | 3837 | 11159 | 126620 | 0.0103 | |
| Qwen2.5-0.5B (HF) | Krackan Point | bf16 | — | 328 | 490 | 831 | 1493 | — | — | 0.0071 | |
| Qwen2.5-1.5B (HF) | Krackan Point | bf16 | — | 638 | 1142 | 2158 | 4352 | — | — | 0.0048 | |
| Qwen2.5-3B (HF) | Krackan Point | bf16 | — | 1140 | 2049 | 4028 | 8156 | — | — | 0.0351 | |
| Qwen2.5-3B-Instruct (FLM, HF) | Krackan Point | Q4_0 | — | 293 | 542 | 1233 | 2996 | 7845 | 73936 | 0.0035 | |
| Qwen2.5-7B-Instruct (FLM, HF) | Krackan Point | Q4NX | — | 565 | 1096 | 2197 | 4961 | 12383 | 107506 | 0.0472 | |
| Qwen2.5-7B-Instruct (FLM, HF) | Strix Halo | Q4NX | — | 582 | 1046 | 2192 | 4970 | 12350 | 107296 | 0.0472 | |
| Qwen3.5-0.8B (FLM, HF) | Krackan Point | Q4NX | — | 179 | 327 | 650 | 1353 | — | — | ||
| Qwen3-0.6B (HF) | Krackan Point | bf16 | — | 383 | 667 | 1246 | 2660 | — | — | 0.0130 | |
| Qwen3-1.7B (HF) | Krackan Point | bf16 | — | 526 | 917 | 1795 | 3748 | — | — | 0.0170 | |
| Qwen3-4B (HF) | Krackan Point | bf16 | — | 1344 | 2382 | 4733 | 10085 | — | — | 0.0161 | |
| Qwen3-4B (FLM, HF) | Krackan Point | Q4NX | — | 376 | 762 | 1835 | 5068 | 15339 | 184476 | 0.0033 | |
| Qwen3-8B (FLM, HF) | Krackan Point | Q4NX | — | 571 | 1187 | 2660 | 6570 | 18318 | 196089 | 0.0030 | |
| Qwen3-8B (FLM, HF) | Strix Halo | Q4NX | — | 570 | 1183 | 2658 | 6588 | 18258 | 195830 | 0.0030 | |
| SmolLM2-1.7B (HF) | Krackan Point | bf16 | — | 406 | 747 | 1467 | 3047 | — | — | 0.0047 | |
| SmolLM3-3B (HF) | Krackan Point | Affine Q4 | — | 296 | 549 | 1230 | 2975 | — | — | 0.0043 |
— not swept at this length, or an expected failure.
Image + text: time to first token vs image tokens (ms)¶
One image and the prompt "Describe this image.". Each budget column is the number of image tokens the image may take; the cell is the time to first token, which is the vision encoder plus the prefill. Decode is generation after the image, the range over the budgets.
| Model | Hardware | Weights | Number of image tokens | Decode (tok/s) | Measured | Verify | ||||
|---|---|---|---|---|---|---|---|---|---|---|
| 70 | 140 | 280 | 560 | 1120 | ||||||
| gemma-4-E2B-it (FLM, HF) | Krackan Point | Q4NX, bf16 vision | 323 | 457 | 811 | 1607 | 4274 | 34.9–36.0 | 2026-11-15 | |
| gemma-4-E4B-it (FLM, HF) | Krackan Point | Q4NX, bf16 vision | 371 | 583 | 961 | 1842 | 4665 | 17.3–18.4 | 2026-11-15 | |
Speculative decoding¶
A draft model proposes a block of tokens and the target model verifies the block in one pass. Target only is the same target model decoding one token per pass. Both decode the same 20 prompts, 4 from each of gsm8k, math500, humaneval, mbpp, mt-bench, greedy, up to 2048 new tokens, prefill excluded. Prompts are 20–305 tokens. Throughput is each engine's total generated tokens divided by its total decode time over the requests in the row. The two engines use different matrix arithmetic (single-token GEMV vs batched verification), so their greedy outputs diverge and the generated token counts differ.
| Model | Hardware | Dataset | Target only (tok/s) | Speculative (tok/s) | Speedup | Tokens / pass | Acceptance | Verify |
|---|---|---|---|---|---|---|---|---|
| Llama-3.1-8B-Instruct (HF) draft: LLaMA3.1-8B-Instruct-DFlash-UltraChat, block 8 | Krackan Point | gsm8k | 11.60 | 27.23 | 2.35× | 4.18 | 0.45 | |
| math500 | 9.61 | 21.93 | 2.28× | 3.39 | 0.34 | |||
| humaneval | 9.57 | 31.10 | 3.25× | 4.96 | 0.57 | |||
| mbpp | 9.61 | 30.15 | 3.14× | 4.62 | 0.52 | |||
| mt-bench | 9.61 | 17.83 | 1.86× | 2.76 | 0.25 | |||
| all 20 | 10.02 | 24.29 | 2.42× | 3.77 | 0.40 | |||
| Qwen3-4B (HF) draft: Qwen3-4B-DFlash-b16, block 8 | Krackan Point | gsm8k | 17.99 | 80.85 | 4.49× | 5.25 | 0.61 | |
| math500 | 17.86 | 90.87 | 5.09× | 6.09 | 0.73 | |||
| humaneval | 17.92 | 77.91 | 4.35× | 5.21 | 0.61 | |||
| mbpp | 17.99 | 72.46 | 4.03× | 4.70 | 0.53 | |||
| mt-bench | 17.94 | 32.22 | 1.80× | 2.10 | 0.16 | |||
| all 20 | 17.92 | 65.44 | 3.65× | 4.31 | 0.48 | |||
| Qwen3-4B (HF) draft: Qwen3-4B-DFlash-b16, block 16 | Krackan Point | gsm8k | 17.97 | 83.00 | 4.62× | 7.04 | 0.41 | |
| math500 | 17.81 | 102.04 | 5.73× | 9.10 | 0.54 | |||
| humaneval | 17.91 | 77.81 | 4.34× | 6.85 | 0.39 | |||
| mbpp | 17.96 | 68.45 | 3.81× | 5.85 | 0.32 | |||
| mt-bench | 17.88 | 27.93 | 1.56× | 2.37 | 0.09 | |||
| all 20 | 17.89 | 63.88 | 3.57× | 5.51 | 0.30 | |||
| Qwen3-8B (HF) draft: Qwen3-8B-DFlash-b16, block 8 | Krackan Point | gsm8k | 11.41 | 33.77 | 2.96× | 5.69 | 0.67 | |
| math500 | 10.18 | 33.76 | 3.32× | 5.89 | 0.70 | |||
| humaneval | 9.50 | 29.61 | 3.12× | 5.13 | 0.59 | |||
| mbpp | 9.42 | 27.14 | 2.88× | 4.52 | 0.51 | |||
| mt-bench | 9.55 | 13.34 | 1.40× | 2.26 | 0.18 | |||
| all 20 | 9.97 | 25.99 | 2.61× | 4.45 | 0.49 | |||
| llama31_8b_dflash_b16 | Krackan Point | gsm8k | 12.00 | 21.36 | 1.78× | 4.07 | 0.20 | |
| math500 | 9.72 | 18.11 | 1.86× | 3.48 | 0.17 | |||
| humaneval | 9.47 | 27.42 | 2.90× | 5.46 | 0.30 | |||
| mbpp | 9.40 | 24.82 | 2.64× | 4.72 | 0.25 | |||
| mt-bench | 9.29 | 14.08 | 1.52× | 2.71 | 0.11 | |||
| all 20 | 10.01 | 19.93 | 1.99× | 3.85 | 0.19 | |||
| qwen3_8b_dflash_b16 | Krackan Point | gsm8k | 11.35 | 33.65 | 2.96× | 7.32 | 0.42 | |
| math500 | 9.97 | 37.82 | 3.79× | 8.60 | 0.51 | |||
| humaneval | 9.25 | 31.07 | 3.36× | 7.03 | 0.40 | |||
| mbpp | 9.38 | 27.14 | 2.89× | 5.85 | 0.33 | |||
| mt-bench | 9.36 | 10.88 | 1.16× | 2.38 | 0.09 | |||
| all 20 | 9.79 | 25.25 | 2.58× | 5.60 | 0.31 |
Vision-Language-Action policies¶
Not autoregressive: these emit one action chunk per observation rather than a token stream, so there is no decode throughput and no context to sweep. Action chunk is the latency to the model's first usable output. Prefix is the token count the backbone attends over.
| Model | Hardware | Prefix | Action chunk (ms) | Measured | Verify |
|---|---|---|---|---|---|
| smolvla_base (HF) | Krackan Point | 241 | 609.1 | 2026-11-15 | |
| smolvla_base, all stages on NPU (HF) | Krackan Point | 241 | 378.3 | 2026-11-15 |
Speech-to-text¶
Each request transcribes one audio clip of at most 30 s. TTFT is from the audio to the first token; Decode is the generation throughput after it; Real-time factor is the request's time over the audio's duration (lower is faster).
| Model | Hardware | TTFT (ms) | Decode (tok/s) | Real-time factor | Measured | Verify |
|---|---|---|---|---|---|---|
| whisper-large-v3-turbo (FLM, HF) | Krackan Point | 540.5 | 78.31 | 0.0881 | 2026-11-15 |
📈 Performance history over time — per-nightly TTFT and decode throughput plotted per model.
Last updated 2026-11-15 · mlir-air 2e310f4 · mlir-aie 1631e1b · llvm-aie 22.0.0.2026102001+4f5df13d