A 60k-token context comes back off disk in 17 seconds instead of 65 seconds of prefill

time to first token, same ~60k prompt, same server

cold prefill  ████████████████████████████████████████████████████  64.6 s
from disk     █████████████·······································  16.5 s   3.9x

…and it still does that with eight sessions restoring at once, with no stalls.

Out-of-tree fixes and hardening for vLLM's OffloadingConnector, developed against vllm 0.1.dev20051+g487ecf187 serving GLM-5.3-Flash (Glm5Next, EXL3 4bpw) at 1M context, TP=2 across two NVIDIA DGX Spark (GB10) hosts.

The KV cache tier lives in a sparse file on local NVMe rather than in RAM, so it can be larger than the GPU pool it backs. Everything loads through vLLM's supported extension points as idempotent start-time source patches. vLLM itself is never forked.

Twenty sessions of ~60k tokens, each evicted by pushing 2.00M tokens of filler through the GPU pool so a revisit genuinely comes back off disk, then restored in concurrent waves. Every session carries five needles at 10/30/50/70/90% depth and a session-unique prefix, so a truncated restore and a cross-session mix-up are distinguishable from each other.

20 of 20 restores correct. No stalls, no timeouts, no tier pressure at any rung.

The comparison that matters is N concurrent restores against N concurrent cold prefills of the same size, so that was measured too — N unique-from-token-0 documents fired at once, with cached_tokens == 0 asserted on every request so nothing can be a cache hit:

Aggregate prefill throughput is flat within 5.8% from N=1 to N=8. Concurrency buys nothing on the prefill side — the GPU is already saturated by a single chunk, so extra concurrent prefills reorder work without adding any. That is the fact which makes the comparison legitimate.

restoring N sessions vs prefilling the same N, same size, same boot

N=2   restore ███·················   18.0 s     prefill ████████████████████  122.5 s   6.8x
N=4   restore ██··················   30.3 s     prefill ████████████████████  250.3 s   8.3x
N=6   restore ████················   69.6 s     prefill ████████████████████  373.9 s   5.4x
N=8   restore ███·················   80.1 s     prefill ████████████████████  503.0 s   6.3x

Call it 5–8×, not a point value. Two runs of the identical configuration put N=6 at 47.3 s and 69.6 s — a 47% spread — while N=2, N=4 and N=8 reproduced within 8% (18.7/18.0, 32.8/30.3, 75.6/80.1). Quoting a single rung to two significant figures would be overfitting to one run. The machine was also not idle during the second run: a harness was serving live traffic against it, which is realistic but not controlled.

A single restore against a cold prefill of the same prompt, same server:

The 6,240 tokens that are re-prefilled are the Mamba alignment tail: the restorable prefix rounds down to an aligned chunk boundary, leaving a remainder bounded by a constant (≤ 7168 tokens) while the prefill you skip grows with depth. Deeper sessions therefore do strictly better than the 3.9× above.

Measured at: tier 260 GB, GPU KV pool pinned to 1,899,014 tokens (770 usable block ids), max_num_seqs=16. Zero NVRM allocation faults across the entire ramp, host MemFree steady at 1.1–1.3 GiB under eight concurrent deep restores.

vLLM admits a restore only when every KV group's chunks are resident in the primary tier at once. That invariant is what makes the tier's size the binding constraint — and the straightforward way to satisfy it is to make the tier big, which RAM will not allow on a machine whose RAM is already the GPU's.

So the staging region is backed by a sparse file on local NVMe instead of /dev/shm. Four edits, gated on GLM53_OFFLOAD_MMAP_DIR: point the region at a file, skip the /dev/shm free-space check, never pre-fault it (a sparse file would otherwise materialise in full), and skip cudaHostRegister on a mapping that is no longer pinnable host RAM. Unset the variable and vLLM's /dev/shm behaviour is untouched.

flowchart LR
    subgraph N1["HEAD NODE — DGX Spark GB10, rank 0"]
        direction TB
        G1["GPU KV pool<br/><b>1.90M tokens</b><br/>20.97 GB pinned"]
        C1["Primary tier<br/><b>260 GB sparse file, NVMe</b><br/>4785 rows · LRU"]
    end
    subgraph N2["PEER NODE — GB10, rank 1"]
        direction TB
        G2["GPU KV pool<br/>rank 1"]
        C2["Primary tier<br/><b>own region, own file</b>"]
    end
    G1 -- "store" --> C1
    C1 == "restore" ==> G1
    G2 -- "store" --> C2
    C2 == "restore" ==> G2

    classDef gpu  fill:#dbeafe,stroke:#3b82f6,stroke-width:2px
    classDef disk fill:#fed7aa,stroke:#ea580c,stroke-width:2px
    class G1,G2 gpu
    class C1,C2 disk
    style N1 fill:#f6f8fa,stroke:#d0d7de,stroke-width:1px
    style N2 fill:#f6f8fa,stroke:#d0d7de,stroke-width:1px

Each rank owns its own region and its own file. There is one tier, no promotion step between tiers, and nothing to cascade or mirror between hosts.

This is the part worth stealing even if you take nothing else here.

The GPU pool and the tier are both LRU over the same stream of blocks. A tier smaller than the pool therefore holds a strict subset of what the pool already has — the GPU answers first, and the tier can never be the thing that serves a restore. Under-sizing it is not a smaller cache, it is no cache, and it fails silently: stores still succeed and every store-side metric still climbs while no restore ever serves.

260 GB tier, restorable context under each reading of the row layout

GPU KV pool         1.90M  ████████
3 rows per segment  5.72M  ████████████████████████
5 rows per segment  3.43M  ██████████████
                           every reading clears the pool — which is the point

Capacity in tokens is num_blocks × (3584 / rows_per_segment). num_blocks is exact: cpu_bytes_to_use // aligned_kv_bytes_per_chunk, and vLLM then creates a region of exactly num_blocks × aligned_kv_bytes_per_chunk. On this stack that row is 54,329,344 B, confirmed twice — 32 GiB → 632 rows, 260 GB → 4785 rows, both exact.

rows_per_segment is bounded, not measured: 1 row/segment is ruled out, since a segment's measured store exceeds what one row can hold, leaving 3–5 here. Size the tier so it clears the pool under the least favourable reading and the ambiguity stops mattering.

In the run above the tier absorbed 3.20M tokens (1.20M seeded + 2.00M filler) with the oldest session still restoring in full, which is what tightens the bound to rows_per_segment ≤ 5.36. That cost 88 G of actual disk per node — well under the address space reserved, because unused row tails stay sparse holes.

The boot-time capacity log explains the offload footprint per token:

ids per 3584-token cached segment across groups: 38
per group: [1, 0, 1, 1, 1, 1, 33]
                              ^^ group 6, SlidingWindowSpec, window=2048

33 of the 38 GPU block ids per segment belong to one sliding-window group — which the offload filter excludes, since a 2048-token window is not worth shipping. Five ids per segment are what actually reach the tier. Any capacity estimate reasoning from total GPU block ids will be wrong by ~7.6×.

Defect 2 produces wrong behaviour with no error at all. Exact code sites and suggested fixes are in UPSTREAM_ISSUE.md.

Fixing 4 also halved the on-disk footprint — 51.81 MiB → 25.91 MiB per region row.

Each of these produced a wrong conclusion before it was caught:

And one label: kv_offload_tiering_* tags tiers tier="<idx>:<ClassName>", not tier="<idx>". With a single tier everything is tier="0:primary", so any counter written to exclude the primary reports zero disk activity for a run that is entirely disk-served. The transfer is visible instead as kv_offload_total_bytes_total{transfer_type="CPU_to_GPU"} — 7433 MiB over 16 load jobs at N=8.

Both patches are idempotent, sentinel-guarded, and fail closed if upstream drifts: an anchor that no longer matches exactly once aborts the patch rather than half-applying it.

// kv_transfer_config — one tier, and it is the disk
{"kv_connector": "OffloadingConnector", "kv_role": "kv_both",
 "kv_connector_extra_config": {
   "spec_name": "TieringOffloadingSpec",
   "cpu_bytes_to_use": 260000000000,   // must exceed the GPU KV pool
   "eviction_policy": "lru"}}

Apply the patches at container start, before the engine imports vLLM, and set:

GLM53_OFFLOAD_MMAP_DIR=/kvoffload              # a directory on local NVMe, not /dev/shm
GLM53_OFFLOAD_GROUP_FILTER=1
GLM53_OFFLOAD_REGION_COPIES=<GPUs per node>    # see note below

The region file must also be reachable at the same path inside every rank's container, and each rank creates its own.

• Run with expandable_segments:True and --enable-cumem-allocator. vLLM drops PYTORCH_CUDA_ALLOC_CONF when a KV connector is configured unless the cumem allocator is explicitly enabled; on unified memory, running without expandable segments is what turns a deep request into a host-memory collapse.
• GLM53_OFFLOAD_REGION_COPIES sizes region rows by the workers that actually share a region. tiering/spec.py picks a worker's slot as torch.accelerator.current_device_index() % world_size, which is the global rank only when all workers sit on one node; with one GPU per host it is 0 everywhere, so every worker uses slot 0 while rows are still sized for world_size. Opt-in and clamped — a value below the real GPUs-per-node would make co-located workers share a slot and corrupt each other.
• Page cache competes with CUDA on GB10. cuda_free tracks MemFree, not MemAvailable, so a tier busy enough to fill page cache can starve an allocation that would otherwise succeed.
• Offload engages from cache turnover, not from pool saturation. BlockPool.free_blocks appends hashed blocks to the back of the free queue and get_new_blocks pops from the front, so every allocation recycles the oldest cached block regardless of how full the pool is. Eviction is paced by allocation rate, not occupancy. The useful sizing question is not "will my working set exceed the pool" but "how much prefix re-use is my workload losing to turnover".

python3 tests/test_mmap_overlay.py        # the patch chain vs pristine upstream sources
python3 probes/test_w31_probe.py          # the concurrency probe's own scoring
python3 probes/test_mamba_probe.py

These need only a Python 3 interpreter. test_mmap_overlay.py applies the edits to real vLLM sources — point it at a copy with PRISTINE=/path/to/site-packages/vllm, and it skips with a clear message if absent. It checks anchors, compilation, idempotence, the disabled-path no-op and sentinel collisions.

They are not decoration. This suite caught real defects before any container start, including an anchor that matched a comment the patch itself had inserted — which silently skipped an edit and produced a server that booted fine and offloaded nothing.

• Restore depth is bounded by Mamba align. The restorable prefix rounds down to an aligned chunk boundary, leaving ≤ 7168 tokens to re-prefill.
• The throughput figure is a range, 5–8×, and one rung is unstable. N=6 came out at 47.3 s in one run and 69.6 s in another under identical configuration — a 47% spread — while N=2, N=4 and N=8 reproduced within 8%. Do not read any single rung as precise, and note the second run shared the machine with live traffic.
• Measured to N=8 concurrent. Beyond that is untested; there was no tier pressure at N=8 (zero allocation retries), so the next limit is likely max_num_seqs rather than the tier.
• Deep single restores are reported at ~60k here. The alignment tail is a constant, so deeper sessions should do better, but this repository does not claim a number it has not measured.
• rows_per_segment is bounded to 3–5, not pinned. Sizing against the least favourable reading makes it moot; pinning it would need a tier shrunk until restores start failing.
• A residual storage overhead remains: rows are uniform across groups, so a small MLA payload still occupies a full row. On a sparse file the unused tail costs address space rather than disk, but it does consume a row, and rows are what LRU evicts.

The disk-backed design is not ours:

• MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks#58 (@MiaAI_lab) — the disk-backed staging region, the residency invariant stated plainly enough to explain a stall we had been chasing from the wrong end, and the observation that under-sizing the tier fails silently with every store-side metric still climbing.
• The Reederey87 GLM-5.3-Flash EXL3 serving kit (@Reederey) — the serving recipe this entire stack runs on: the two-node TP=2 bring-up, the overlay patch mechanism (idempotent, sentinel-guarded, fail-closed) that every patch here follows, and the boot-time capacity log that produced the block-id breakdown above.

The group and row-sizing fixes, the upstream defect analysis, and the measurement harnesses in probes/ are this repository's own.

Apache-2.0. Not affiliated with the vLLM project.