vllm.cpp reads a number of environment variables. Most are kernel-internal tuning and bisect knobs that belong to the engineering record, not to a deployment; this page documents the subset that changes user-visible behavior or is a supported operational lever. Everything else is enumerated on the kernel-internal allowlist (scripts/env-doc-allowlist.txt) and kept honest by a CI check (see Keeping this reference honest).

Unless stated otherwise, a flag-style knob is read as on when its value is a non-empty, non-0/false/off string, and the listed default applies when the variable is unset. None of these are required to run the engine.

The weight-residency knobs VT_GGUF_MMAP, VT_GGUF_PREFAULT, VT_MOE_EXPERT_STREAM, VT_MOE_EXPERT_STREAM_SLOTS, VT_MOE_EXPERT_STREAM_SLOT_BYTES and VT_DEVICE_WEIGHT_BUDGET_BYTES, and the four placement knobs VT_PLACEMENT_OVERRIDES, VT_CPU_MOE, VT_N_CPU_MOE and VT_PLACEMENT_FIT, can also be set in the server's JSON config, under the vllm_cpp key of --offload-config. That is the documented deployment surface; these variables are the override:

environment variable  >  --offload-config's vllm_cpp key  >  built-in default

VT_CPU_MOE, VT_N_CPU_MOE, VT_PLACEMENT_OVERRIDES and VT_PLACEMENT_FIT are ENG-HYBRID-PLACEMENT (#149). They are read, validated, desugared, resolved to a per-layer decision, and acted on: a placed layer runs its routed experts on the placement device, with the hidden state crossing at the layer boundary.

Three limits, and each of them bites.

Five architectures, not all of them. Qwen3MoeForCausalLM, Qwen3.5/3.6, Nemotron-H, DeepSeek-V2 and Kimi-Linear route through one shared seam and honour these variables. The rest do not, for reasons that differ and are worth telling apart: Laguna runs its expert GEMMs on the device but presents a per-token host-float FFN boundary, so it has no [T,H] block to hand the seam; Gemma4's expert path accumulates into a caller's buffer rather than returning one; DeepSeek-V4 runs its experts on the host, where a placement has nothing to move; and GLM-5-Next, dots3-note, Kimi-K3 and qwen4_exp have no reachable MoE forward yet. On all of those the variables are silently inert.

The TARGET is the CPU. The engine may run on any device; the destination may not. A placement naming an accelerator is refused at startup, because the queue is created once for the process and a non-CPU queue would leak its stream.

No model has been run with a placement in force. The mechanism is gated by unit and round-trip tests — the round trip is byte-identical to computing in place, and mutation-proven — and the end-to-end token gate is owed. Treat a resolved placement on the install line as what was asked for, not as a measured result.

docs/FEATURES.md carries the capability as partial for these reasons and is the row to trust for scope.

The polarity is deliberate. These variables exist so a benchmark arm is switchable without restarting the server with a new document, so an override that could not turn a configured knob back off would be useless. VT_X=0 therefore beats a config true, and the engine prints one line at startup naming the document it installed and a second naming every variable that would WIN over a field of it — a configuration silently overridden by something exported weeks ago is the one way this precedence hurts. "Would win" is exact for the three boolean knobs, where any value wins. For the two COUNTS it is narrower than presence: an empty, garbage or non-positive value falls through to the config under the tolerant parse those readers have always used, so VT_MOE_EXPERT_STREAM_SLOTS=banana overrides nothing and is not reported as an override.

VT_DEVICE_WEIGHT_BUDGET_BYTES is narrower again, and in a different direction. Its grammar is decimal digits only — no sign, no space, no trailing text — so a value outside it falls through to vllm_cpp.device_fit.weight_budget_bytes and then to the device probe. Inside that grammar every value wins, 0 included: 0 is this knob's suppression spelling rather than an absent one, so it is an override and is reported as one. It joined the config surface after the other five (#1127).

See the expert streaming guide for the config form and .agents/specs/weight-residency-config.md for why the extension is a namespaced key rather than a field on vLLM's mirrored OffloadConfig.

VT_MOE_EXPERT_STREAM_STATS_EVERY is deliberately not a config key. Every knob above changes what memory the process reserves or where a weight lives, which is a deployment decision; that one changes only how often a diagnostic line reaches stderr. It is the instrument, not the configuration.

These change how the engine runs and have no CLI flag (or complement one).

Behavior of the GGUF weight loader (CPU path). See the quantization support table for the current format and backend coverage.

Opt-in host-side residency cache for routed MoE experts on the CPU keep-quant path, used when a mixture-of-experts GGUF is served from a borrowed (mmap'd) weight tower larger than RAM — the Qwen3.5-family MoE decode path in src/vllm/model_executor/models/qwen3_5.cpp. Borrowing is what makes such a model fit; this cache is an attempt to make it fast, by serving each expert slice out of a fixed array of equal-sized slots filled by one contiguous copy instead of letting the kernel demand-page the mapping in router order.

Off by default, and honestly still experimental. The W4 measurement on Qwen3.8-2.4T-A95B UD-Q1_0 (.agents/specs/expert-streaming.md, "W4 measured") is token-identical output and a 4.5x TTFT improvement (3318 s to 733 s) with no steady-state decode gain at all, because the slot is filled by copying from the mapping and so still takes the page fault it was meant to avoid. Turn it on to reproduce or to work on that; it is not yet a deployment recommendation.

The two budget knobs are read once, when the cache is first constructed, and only if streaming is on. Host memory reserved is slots x slot_bytes, allocated up front and never grown — the engine prints the resolved values as [expert-stream] ON slots=... slot_bytes=... resident=... GiB on stderr.

The Kolibri-1 CPU decode path interleaves the weight-arena allocation and the forward pass over every NUMA node the process may use (MPOL_INTERLEAVE, src/vllm/model_executor/models/kolibri1_numa.h, wired at kolibri1_weights.cpp in LoadKolibri1Weights). Multi-node hosts otherwise keep the arenas on the loading node's memory, and 32-thread decode then reads most weight bytes remotely. Measured on the 4-node aarch64 host (docs/bench-evidence/kolibri1-numa-interleave-20261009.md): 4.19 to 4.97 decode tok/s at 32 threads with the 16 GiB dequant cache (+18.6%), prefill −14%, golden decode chain unchanged. The guard saves and restores the caller's mempolicy, and it is inert on single-node hosts and on Windows. The lever composes with the dequant cache; with the cache off and 8 threads its effect collapses to ~2%, because that path is allocation-bound.

Default-on fast paths, each with an off switch. They exist so a suspected kernel bug can be bisected without a rebuild: set the variable to 0 to fall back to the portable/reference path. In normal operation leave them unset.

| VT_ADOPT_DEVICE_BYTES | on (acts only where the backend advertises host-addressable device memory: Vulkan always, Metal and ROCm where the part reports unified memory. MEASURED on Vulkan only) | After a weight is uploaded, re-point its host buffer AT the device allocation instead of keeping a second copy. On a unified box the two copies come out of the same RAM. MEASURED on GB10, Qwen3.6-27B bf16 (50.89 GiB): with the mirror the process reaches VmRSS 100.759 GiB and drives the machine to MemAvailable 13.85 / MemFree 1.13 GiB of 119.6 GiB before it has even finished allocating; without it the same load completes at VmHWM 53.413 GiB. Qwen3-4B: 16.392 -> 9.607 GiB. Vulkan allocation is byte-identical either way. 0 is the same-binary A/B back to the two-copy behaviour, and it moves ONLY this lever: the VT_LOAD_DIRECT_UPLOAD post-upload source-page release still runs at 0 and on a backend without host-addressable device memory, so the two knobs stay independent. It is an adoption, not a release — the bytes survive at the device address and every reader sees them — so tokens are unchanged either way (test_opt_paged_engine on Vulkan is 6/6 token-exact, 96/96, both arms). REACH AND MEASUREMENT ARE NOT THE SAME SET, and this row used to read as if they were. The lever is gated on vt::Backend::DeviceMemoryIsHostAddressable(), read at both AdoptDeviceBytesAsHost branches in src/vllm/model_executor/models/qwen3_5_weights.cpp. That predicate defaults to false (include/vt/backend.h), so a backend has to opt in, and THREE now do. VulkanBackend (src/vt/vulkan/vulkan_backend.cpp) returns true unconditionally, because every allocation it makes is HOST_VISIBLE|HOST_COHERENT and persistently mapped. Since cffe59b02 (#1477, which moved the portable reference tier off the WIDER UnifiedMemory()) MetalBackend (src/vt/metal/metal_backend.mm) answers MetalContext::unified_memory(), i.e. dev.hasUnifiedMemory, and RocmBackend (src/vt/rocm/rocm_backend.hip) answers its own unified_memory_, i.e. a managed allocator or an integrated part with PageableMemoryAccess. So the lever ACTS on Apple silicon and on an integrated ROCm part, and nobody has measured it on either — every number above is GB10 through Vulkan. Read those two as reachable-and-unmeasured rather than as covered by this row's numbers (#1502). It is still inert on CUDA and CPU, neither of which overrides the default. That is now CHECKED rather than merely observed (#1635), and in two halves, because no CI job has a GPU on which a live CudaBackend could answer: a static_assert beside the class in src/vt/cuda/cuda_backend.cu fails the cuda-fat-build compile if CudaBackend ever declares its own DeviceMemoryIsHostAddressable, so its answer IS the inherited one; and a case in tests/vt/test_backend.cpp pins that inherited default to false on a subclass that declares no override, on every host lane. Do NOT cite tests/vllm/platforms/test_platform.cpp for this: its FakeUnifiedAddressablePlatform reports device_type() == kCUDA but its backend() returns vt::GetBackend(DeviceType::kCPU), so its CHECK_FALSE(gb10.backend().DeviceMemoryIsHostAddressable()) reads the CPU backend. tests/vt/test_cuda_backend.cpp observes the real backend but skips wherever no GPU is present, so it asserts nothing in CI. The CPU backend reports UnifiedMemory() == true while this narrower predicate stays false — the two properties disagreeing is the whole reason the second one exists | | VT_QWEN35_ALIAS_HOST_WEIGHTS | on (Qwen3.5/3.6 family; acts only where the platform advertises host_memory_is_device_addressable() — a GB10-class integrated part today) | Hand a dense weight's HOST bytes straight to the device kernel instead of allocating a device copy and uploading into it. On a part whose kernels can dereference host storage that copy buys nothing and costs a second full resident copy of every dense weight out of the same RAM. MEASURED on GB10, Qwen3.8-2.4T-A95B UD-Q1_0 (369.97 GiB) on --device cuda: with the second copy the load completes at 61.20 GiB resident and then exhausts the 119.631 GiB box inside its FIRST forward, zero decode steps over seven attempts (issue #1299); with the alias the same run reaches 32/32 decode steps at peak RSS 97.75 GiB. The instrument counts 60.793 GiB of dense weight aliased rather than duplicated (first-forward totals, at call 1361, where re-homing plateaus) against ~9.2 GiB of misaligned GGUF borrows that decline and still stage. 0 is the same-binary A/B back to the staging behaviour, and it is not only a bisect lever: src/vllm/model_executor/models/laguna.cpp records a MEASURED GB10 penalty for reading system-allocated memory from the GPU rather than a cudaMalloc allocation, worst on a long-K low-parallelism GEMV, so a decode regression has to be separable from the workload. The substitution is otherwise indistinguishable — the aliased pointer is re-homed to 256 bytes, cuBLASLt's own CUBLASLT_MATMUL_PREF_MIN_ALIGNMENT_A_BYTES default, which dominates every explicit pointer gate in the CUDA kernels (the strictest asks 32). A DISCRETE device answers the predicate false and is byte-for-byte unchanged. No effect on CPU, Vulkan, Metal or XPU, whose platforms do not advertise the property | | VT_QWEN35_STAGE_RESERVE_BYTES | 12 GiB (Qwen3.5/3.6 family; read once, and only by the SAFETENSORS loader) | The device memory a staged model has to leave behind for the KV cache and activations. The retag above is measurably slower than a true device copy on GB10, so a model that FITS should stage; #1299 recorded Qwen3.8-2.4T-A95B exhausting a 119.631 GiB box precisely BECAUSE the CUDA arm paid for its weights twice, so a model that does not fit must not. The rule is 2 * model_weight_bytes + VT_QWEN35_STAGE_RESERVE_BYTES <= device_total_bytes, and the factor of TWO is the substance: on a platform that does not release the host mirror after upload, a staged weight is ADDITIVE. MEASURED on the committed gate (scripts/dflash2-speed-gate.sh, Qwen3.8-27B bf16 + DFlash2 k=7, c=1, one boot, same oracle wheel): retagged 12.361 tok/s -> staged 15.029, against vLLM's 16.363, i.e. ratio 0.7587 -> 0.9185, +21.6%. IT IS DECIDED ONCE, AT LOAD, AND THAT IS THE WHOLE POINT. The first version of this policy asked Backend::DeviceMemoryInfo PER WEIGHT and compared LIVE FREE against a fraction of total; free falls as the host mirror loads, so the floor degraded into "stage the first N GiB, then stop" and captured 8.1% of the available 21.6% (a22030924, gate 13.3625). The rule was already written next door: include/vllm/platforms/interface.h says the budget is "TOTAL rather than FREE, because free at load time carries the page cache and whatever else the box is doing, which would make a load-time verdict a function of contention." Only the safetensors loader latches it, so a GGUF checkpoint never stages and keeps exactly the behaviour #1299 shipped — which also makes the file total a sound proxy, since a bf16 safetensors weight occupies its file size in host RAM while a GGUF one expands on load. Parsed with atoll: unset, empty, unparsable and <= 0 all fall back to 12 GiB rather than refusing, so a typo cannot silently shrink the reserve and double a model's residency. An unknown device total or an unknown model size answers NO. VT_QWEN35_ALIAS_HOST_WEIGHTS=1 pins the retag on regardless; =0 forces staging regardless. LOWER admits bigger models to staging; HIGHER is more conservative | | VT_LOAD_DIRECT_UPLOAD | on | Load a weight the device consumes VERBATIM by VIEWING the safetensors mmap (OwnedBytes::Borrow, keep-alive on the mapping) instead of copying it into an owned host buffer first, so the device upload reads the file mapping and the load moves those bytes ONCE rather than twice. Only whole-range same-size copies qualify — a transpose, a dtype conversion, a dequant, a concatenation or a load-time repack always takes the copy path, and the helper re-checks numel * sizeof(dtype) == span and fails closed to the copy on any mismatch. 0 is the same-binary A/B back to copy-then-upload. Bytes are identical either way, so tokens are unchanged. MEASURED on GB10, Qwen3.6-27B bf16 (50.098 GiB), Vulkan, same binary both arms: the weight-load phase goes 19.27 -> 12.48 s warm (1.54x) and 52.62 -> 32.75 s cold (1.61x), load-and-one-token 30.39 -> 22.47 s warm and 62.98 -> 55.60 s cold, with every ON leg beating every OFF leg. Total bytes MOVED 100.196 -> 81.260 GiB: the host materialization pass drops 50.098 -> 31.162 GiB while the 50.098 GiB device upload is unchanged (the model still has to be uploaded once). 37.8% of this checkpoint qualifies; the rest is merged (qkv, gate_up) or transposed (lm_head) at load and correctly still copies | | VT_LOAD_STATS | off | =1 prints one line per load phase (mmap+header, weights) with its wall time, plus the bytes the load MOVED: host_copy (source bytes materialized into an owned host buffer), borrowed (source bytes viewed in place by the direct-upload path) and device_upload (bytes copied host to device). Diagnostic only; it changes no numerics. Issue #150 | | VT_VULKAN_ALLOC_STATS | off | =1 prints a device-memory line on every 1 GiB high-water crossing and a summary at exit: live buffer count, bytes REQUESTED by the caller, bytes COMMITTED by the driver (VkMemoryRequirements::size), peak live bytes, and the process/system context (VmRSS, VmHWM, MemAvailable, Cached) read from /proc. On a unified-memory device the Vulkan heap IS system RAM, so separating "the backend allocated it", "the process allocated it some other way" and "it is page cache" is the whole of a memory attribution. Diagnostic only; it changes no numerics. The counters themselves are always maintained (one relaxed atomic per allocation) and are readable from a test through vt::vulkan::DeviceAllocStatsSnapshot(). Vulkan-only | | VT_VULKAN_DISPATCH_STATS | off | =1 traces every Vulkan compute submit to stderr (index, shader, workgroup count) BEFORE its fence wait, prints any wait over 200 ms, reports a running dispatch rate every 100 submits, and dumps a per-shader histogram at exit. Printing before the wait is what makes a HANG visible: a post-wait print never runs if the fence never signals, so the last line names the dispatch that hung. This is how the coopmat out-of-bounds load was found. Diagnostic only; it changes no numerics. Vulkan-only | | VT_VULKAN_GEMV_UNROLL | 4 | =1 forces the un-unrolled decode GEMV body. Four independent accumulators keep four reads per lane in flight instead of one -- memory-level parallelism, not instruction count. It rides a specialization constant, so both arms are the same committed module and A/B in one binary. MEASURED 1.055x, 7 of 8 interleaved pairs. Worth noting it measured 5/8 and was REVERTED earlier the same day: that test ran while the GPU was only 26% busy, where a 10% GEMV win moves e2e by 1.4% and is unresolvable against this box's noise. After the ring fix made the run GPU-bound the same code reads 7/8. A negative result is regime-dependent. Vulkan-only | | VT_VULKAN_GEMV_PACK | 2 | Load width for a 16-bit decode GEMV operand: 0 one element per load (2 B), 1 two through the buffer's 32-bit view (4 B), 2 four through a 64-bit view (8 B). Same bytes, same coalescing (32 lanes still cover 128 or 256 CONSECUTIVE bytes), so this cannot reduce DRAM traffic; it reduces LOAD INSTRUCTIONS. MEASURED, and the two regimes disagree by 7x, which is the point. In benchmarks/vulkan_gemv_ab.cpp over the 27B's own decode shapes (9 arms x 4 rotated passes, each arm paired against the width-0 baseline measured IN THE SAME PASS, because the box drifted 15.5% peak-to-peak between passes) width 2 reads 1.086x. In REAL 27B decode it reads 1.012x and 1.020x, two GPU-timestamp two-length diffs (output-len 36 minus 4 over 32 tokens), moving vt_matmul_vec from 90.0% to 91.1% of the 273 GB/s roof. The sweep re-reads one 356 MB buffer 320 times, so its DRAM rows and TLB stay hot and instruction issue is visible; decode streams 50 GB once per token, where DRAM is the whole story. e2e: 4.126 -> 4.157 tok/s, 5 of 6 clean pairs, against a 0.88% clean-leg noise floor. Degrades a width at a time when K or an operand byte offset is not aligned to it, and declines entirely for an f32 operand, which is already one element per 32-bit word. Unlike the row count this axis DOES change the answer's low bits, because it repartitions K across lanes; the opt-125m STRICT gate (6/6 token-exact) is what clears it. Vulkan-only | | VT_VULKAN_GEMV_ROWS | 1 | Output elements per decode-GEMV workgroup, 1, 2 or 4; llama.cpp's NUM_ROWS (mul_mat_vec_base.glsl:90, set per vendor by rm_stdq in ggml-vulkan.cpp:4705-4719). Raising it lets one loaded activation element feed several weight rows. It cannot reduce DRAM traffic either -- the weights are read exactly once per token at any row count -- and the activation re-reads it saves were L2 hits. MEASURED NEGATIVE on GB10 and shipped OFF: 0.966x at 2 rows and 0.968x at 4, in all four passes and at every load width, most likely because halving the workgroup count halves the independent sequential read streams the memory controller sees while each surviving workgroup interleaves rows K*2 bytes apart. Kept as an axis only because llama.cpp raises it exactly on the AMD GCN and Intel parts this backend's portability case is aimed at, and deleting it would mean rediscovering it there. Bit-identical to 1 at any value -- each lane accumulates the same strided subset of K in the same order -- so it is a pure scheduling knob. Vulkan-only | | VT_VULKAN_RING | 128 | Descriptor-set ring depth per pipeline, i.e. how many times one shader may be recorded before the batch must flush. This is what actually caps batch length: at 16 it forced a submit every 40-46 dispatches, because vt_rms_norm runs 112 times per forward pass and exhausted its ring seven times. At 128 a whole forward pass (368 dispatches) fits in ONE submit. Each avoided flush is a vkQueueSubmit plus a blocking vkWaitForFences, and a host profile puts 62% of on-CPU time in the kernel and NVIDIA driver against 14% in our own code -- so submits are the host cost. Lower it to bisect or to cut descriptor-pool size. Vulkan-only | | VT_SERVER_VERBOSE | off | =1 turns on the server's chatty progress output, and is the umbrella switch the individual knobs fall back to: VT_SERVER_PREFILL_PROGRESS reads it when unset, so VT_SERVER_VERBOSE=1 alone enables chunked-prefill progress (rate-limited to ~2 Hz per request). examples/server sets it from its own verbosity flag, so it is user-facing rather than a kernel tuning switch | | VT_ENGINE_STEP_LOG | off | =1 prints an EngineCore step heartbeat to stderr (unfinished count, whether the model executed, output rows, elapsed). Also on under VT_SERVER_VERBOSE=1. It is how you tell an idle engine from a stalled one: a run of model_executed=0 with a non-zero unfinished count is work the scheduler cannot admit. Diagnostic only; it changes no generation | | VT_VULKAN_INFLIGHT | 2 | How many command-buffer submissions may be in flight at once. At 1 a flush submits and BLOCKS on its fence, so all host recording is serialized against an idle GPU; at 2 the flush returns and the next token records while the previous one runs, worth a MEASURED -1.41 ms/token on 27B (5/5 paired legs). Each slot owns a disjoint descriptor-ring slice, so raising this raises descriptor memory | | VT_VULKAN_HOST_PROFILE | off | =1 prints a per-phase HOST time breakdown at exit (submit, record, descriptor update, bookkeeping, pipeline lookup, and time blocked on the GPU). Diagnostic companion to VT_VULKAN_DISPATCH_STATS, which covers the DEVICE side; use it to separate host cost from GPU wait | | VT_VULKAN_BATCH | on | =0 forces the per-dispatch submit-and-wait path instead of recording many dispatches into ONE command buffer (with a pipeline barrier between them) and submitting once. MEASURED on GB10: decode 2.62x, faster in 8 of 8 interleaved pairs (24.8 -> 64.5 tok/s), 40-46 dispatches per submit, opt-125m still 6/6 token-exact. Batching is sound only because all three host-read paths drain the batch first: Backend::Copy and Memset (host memcpy over the mapped allocation), Synchronize, and Backend::FlushPending, which op_provider.cpp calls before running a PORTABLE REFERENCE TIER kernel directly over device memory. Without that third one a host kernel would read stale bytes silently. It exists for the same-binary A/B and as the bisect lever. Vulkan-only | | VT_VULKAN_SMART_BARRIERS | off | =1 records a vkCmdPipelineBarrier before a batched dispatch ONLY where a real dependency exists, instead of before every one. The batched path otherwise emits ~900 full COMPUTE->COMPUTE memory barriers per 27B decode token; llama.cpp instead calls ggml_vk_sync_buffers at 42 explicit call sites and tracks per-scratch-buffer need_sync flags (ggml-vulkan.cpp:3193, :8687-8748 @ pin 237ad9b96). This is that idea made general: each dispatch's READ set and WRITE set come from the committed SPIR-V's NonWritable decorations (vulkan_spirv.h § writable_mask), which glslang enforces, and a barrier is skipped only when none of read-after-write, write-after-write or write-after-read holds against everything recorded since the previous barrier. It DEFAULTS TO EMITTING: buffers are compared whole rather than by byte range, an operand that is both read and written counts as a write, and an unrecognised binding counts as writable, so every uncertainty costs a barrier rather than correctness. Both arms live in one binary because a cross-BUILD comparison of two barrier policies has already produced a false 1.2x in this campaign -- and because a policy that drops a real dependency computes WRONG NUMBERS, which is invisible on llvmpipe and was invisible for a whole benchmark block the last time it happened here. Vulkan-only | | VT_VULKAN_GEMV | on | =0 forces the decode GEMV onto the portable scalar GEMM instead of vt_matmul_vec. The vec tactic gives each output element a WORKGROUP whose lanes stride K, so adjacent lanes read adjacent addresses; the scalar kernel puts one invocation on each output element and loops K there, which for the MatmulBT weight layout leaves adjacent lanes K*2 bytes apart, each pulling its own cache line for 2 bytes of it. Selected only for MatmulBT at M=1 with K at least one workgroup wide -- the other orientation is already coalesced and the vec shape would make it worse. MEASURED on GB10: GEMM ms/call improves in 7 of 8 interleaved pairs, median 1.8x. Its K reduction is a tree, so unlike the scalar kernel it does NOT share the CPU's accumulation order; the opt-125m STRICT gate (6/6 token-exact) is what clears it. Vulkan-only | | VT_VULKAN_RMSNORM | auto | Which vt::RmsNorm SPIR-V module runs: wide forces the 1024-invocation subgroup-reducing one, base forces the portable 128-invocation one, unset lets the device capability decide (1024 invocations on the X axis plus compute subgroup BASIC and ARITHMETIC). The wide module exists because RmsNorm dispatches ONE WORKGROUP PER ROW and a batch-1 decode step has exactly one row: on Qwen3.6-27B that put 128 invocations on a 5120-wide row, four warps of one SM, with the rest of the GPU idle. MEASURED on GB10 by the two-length GPU-timestamp diff: 0.0611 -> 0.0123 ms/call, 7.88 -> 1.59 ms/token, and paired decode 241.9 -> 235.6 ms median TPOT. The tell that it was OCCUPANCY and not the reduction is that the SAME shader costs 0.066 ms/call during PREFILL, where 32 rows give it 32 workgroups and 32x the data. It exists for the same-binary A/B and so the unit gate can exercise the fallback on hardware that would always pick the wide arm. Vulkan-only | | VT_VULKAN_MATMUL_NCOLS | 4 | Output columns each lane of the portable scalar GEMM computes, in the [K,N] (non-transposed) orientation only. At 1 the kernel is the flat one-invocation-per-output-element body; above 1 a workgroup takes 128*NCOLS CONSECUTIVE output columns of one row, so at each step of K it reads a contiguous run of that many elements instead of 128. This is the ONE decode GEMM that cannot reach the vt_matmul_vec tactic, because in [K,N] the lanes are already coalesced and the GEMV shape would make them strided; on the 27B it is the lm_head, m=1 k=5120 n=248320, 2.54 GB moved per token. MEASURED on GB10, 27B decode, ms/call medians over interleaved replicates: NCOLS 1 = 12.48, 2 = 12.46, 4 = 11.54, 8 = 12.81, with 4 winning 6 of 6 interleaved pairs against 1. Blocking is a TRADE, not a monotone win: at 8 the dispatch falls to 243 workgroups (~31k threads) and the device runs out of work to hide memory latency with faster than the longer contiguous run buys back. It rides a specialization constant, so every arm is the same committed module and they A/B in one binary. Every arm is BIT-IDENTICAL -- each accumulator owns one output element and sums the whole K sequentially, which is the CPU kernel's order -- so this kernel keeps the byte-exact tier that the coopmat and GEMV tactics gave up; a memcmp gates that. Vulkan-only | | VT_VULKAN_COOPMAT | on | =0 forces the Vulkan GEMM onto the portable SCALAR kernel instead of the cooperative-matrix (tensor-core) tactic. The coopmat path is selected only where the device reports the exact 16x16x16 bf16/bf16/f32/f32 SUBGROUP configuration, subgroup size is 32, both operands are bf16, and M, N and K are all multiples of 16. The whole-tile requirement on M and N is not a tuning choice: coopMatLoad reads a full 16x16 tile with no masking, so a partial tile reads past the operand and can fault the GPU. Ragged shapes fall back to the scalar kernel; this switch bypasses that selection entirely. It exists for the same-binary A/B in examples/vulkan-gemm-ab (measured 11.1x-32.9x on NVIDIA Thor) and as the bisect lever if a coopmat result is ever suspect. Vulkan-only | | VT_GLM5_NEXT_DEVICE_EXPERTS | off (opt-in) | =1 lets Glm5NextForConditionalGeneration (GLM-5.3-Flash) accept a non-CPU queue and route its routed-expert keep-quant GEMM to the device, against banks made resident by dense_attn::ResidentWeight. IT IS OFF BECAUSE THE PATH IT ENABLES IS MEASURED TO CRASH, not because it is unmeasured. On dgx:gpu0 against the published 101.24 GiB UD-Q2_K_XL artifact, ALL THREE --device cuda legs died with SIGSEGV (rc=139) having emitted no token, interleaved against three --device cpu legs that all emitted Paris. from the same binary. The mixed-residency reading of those legs is FALSIFIED and the cause is elsewhere: the two log lines that suggested it are once-flags, and the process dies in StoreCaches, which host-stores into the runner's cudaMalloc KV pages after the forward has already returned. That defect is older than this knob and only became reachable when the non-CPU refusal above it was removed; see .agents/specs/glm5-next-flash.md O49 and #2480, which owns the fix. The default is the refusal the tree carried before the arm existed, because turning a clean named error into a segfault is strictly worse for a user. Set this only to debug that crash; it is not a serving knob. Parsed strictly (1 and nothing else, not the usual first-character rule) precisely because it opts into a crashing path. Inert on every other model and on --device cpu. See .agents/specs/glm5-next-flash.md O46 and #2464 | | VT_GLM5_NEXT_DEVICE | off (opt-in) | =1 routes the entire GLM-5.3-Flash forward through Glm5NextDeviceForward, which dispatches embedding, RMSNorm, MoE combine, the k-pool indexer and lm_head through vt::* device ops on the queue, keeping MLA attention, mHC sites and the dense MLP as host-fallback islands (the kimi_linear_device.cpp single-queue pattern). This is a superset of VT_GLM5_NEXT_DEVICE_EXPERTS: when on, the whole forward delegates and the per-arm experts split is not reached. On a CPU queue the vt::* kernels use float32 accumulation where the host reference uses double, so the output agrees within a float-vs-double envelope rather than byte-exact. Inert on every other model. See .agents/specs/glm5-next-flash.md W9c-3 and #3175 |

Read-only observability; none change output.

Discrete-ROCm (gfx1201) bring-up knobs from PR #140. Off by default; no effect on CUDA/CPU builds beyond the documented behavior.

VT_GEMMA4_MLP_MOE_PARALLEL used to be listed here with a default and an effect, and it was never wired: it existed only in the deferred layer-loop path that PR #140 did not ship, so nothing in the tree has ever read the name. Setting it does nothing, and the row is gone rather than caveated (#2389).

The remaining variables (roughly 120 VT_GDN_*, VT_FP4_*, VT_FP8_*, VT_MOE_*, VT_ATTN_*, VT_NVFP4_*, VT_FUSE*, and similar) are kernel implementation and micro-tuning switches. They are not part of the deployment surface, their meaning is tied to a specific kernel, and they are recorded in the engineering ledger (.agents/parity-ledger.md) and docs/BENCHMARKS.md where the A/B that introduced each one lives. They are enumerated on scripts/env-doc-allowlist.txt so the CI check below can tell a known kernel-internal knob from a new, undocumented one. If you need to understand one, grep its name under src/ for the read site and the ledger for the measurement.

scripts/check-env-doc.py runs in both directions, and a knob has to satisfy both.

Read but not documented. It scans every VT_* / VLLM_* environment name read from src/ and include/ and fails if any name is neither documented on this page nor listed on the kernel-internal allowlist (scripts/env-doc-allowlist.txt). A newly-introduced production env var therefore cannot land silently: the author must either document it here or classify it kernel-internal on the allowlist.

Documented but read by nothing. Every variable in a table on this page must be read by at least one file CMake compiles (src/, include/, examples/), and a mention in a comment does not count, so deleting the last reader of a knob fails the gate instead of leaving its row here promising behaviour the tree does not implement (#2389). That is the direction the page was missing when VT_QWEN35_STAGE_MIN_FREE_FRAC and VT_GEMMA4_MLP_MOE_PARALLEL outlived their implementations. A knob that is legitimately unread carries a stated reason in the checker's UNREAD_EXCEPTIONS, which is itself gated: an entry with no reason, or one whose variable has since left the tables or gained a reader, fails the check.

The check runs in CI (the agent-record job) and has a unit/mutation test at tests/scripts/test_check_env_doc.py.