Goal

Extend the existing WARM UP SELECT statement with PROPERTIES ("read_index_only" = "true") to call Lance SDK index prewarm on the BEs used by subsequent queries. Reduce query-time random S3 reads for index contents by loading the index into the same BE-local shared cache used by normal queries.

Parent tracker: #66340. Completion on branch-4.1 follows the parent tracker convention; this is not a commitment to a published release date.

Scope

The statement waits for all target BE prewarm calls to complete and returns success or an explicit error directly to the client.

The implementation selects logical indexes from the projected columns (* selects all indexes), including all active segments in one pinned dataset snapshot. It reuses the SDK's existing prewarm implementation and Doris's shared Lance session.

SQL interface

-- Prewarm all indexes in the current execution context.
WARM UP SELECT * FROM lake_catalog.demo.items
PROPERTIES ("read_index_only" = "true");

-- Prewarm indexes on the selected columns.
WARM UP SELECT embedding FROM lake_catalog.demo.items
PROPERTIES ("read_index_only" = "true");

-- Existing data prewarm behavior is unchanged.
WARM UP SELECT * FROM lake_catalog.demo.items;

• Reuse the existing PROPERTIES clause and WarmupSelectCommand; do not introduce a separate WARM UP INDEX statement.
• Omitting read_index_only, or setting it to false, retains the existing data-scan/file-cache warmup path. Reject unknown properties and invalid boolean values.
• In index-only mode, read necessary metadata and index contents through the SDK; do not scan or materialize table data. The cache destination is the shared Lance index cache, so this mode does not require the data-file cache to be enabled.
• * selects all logical indexes; a column list selects indexes associated with those columns. Validate every column and reject a selection with no matching indexes. Deduplicate logical indexes with multiple segments or repeated projected columns.
• The first implementation supports Lance catalog tables. Reject unsupported tables, WHERE, and EXPLAIN in index-only mode explicitly; do not silently fall back to scanning data or ignore a filter.
• Use the current session's compute group. To warm another group, select that group in the session first. In deployments without compute groups, use the eligible BEs selected by existing query/resource-tag policies.
• Warm all eligible target BEs in that execution context, without including unrelated groups.
• Resolve one dataset snapshot and target BE set for the entire statement. Workers must not independently resolve latest; all selected indexes use the same snapshot and deadline.
• Retain ADMIN and table SELECT checks, plus compute-group USAGE in cloud mode, before prewarm IO.
• An unsupported index, inaccessible storage, or failed target BE causes an explicit statement error. Include failed BE/index context without credentials or sensitive storage options.
• Return results only after every selected index completes on every target. Return one row per logical index: table, index, pinned dataset version, target BE count, and elapsed milliseconds. Binary PREPARE and EXECUTE must advertise the same schema.
• The statement uses SQL/RPC timeouts and observes query/connection cancellation. A timeout does not roll back cached entries or guarantee immediate interruption of SDK IO. Reruns must not inherit cancellation from an earlier prepared-statement execution.

Implementation

WARM UP SELECT ... PROPERTIES ("read_index_only" = "true")
  -> WarmupSelectCommand selects the index-only execution path
  -> FE validates access and resolves one snapshot, selected indexes, and target BEs
  -> FE sends bounded synchronous prewarm RPCs and waits for completion
  -> BE opens the fixed snapshot through LanceSessionManager
  -> lance-c calls dataset.prewarm_index(index_name)
  -> later queries reuse the same BE-local session/index cache
  -> FE returns success or an explicit error

Lance-C: a thin binding

The upstream binding provides the following public entry point:

int32_t lance_dataset_prewarm_index(
    const LanceDataset* dataset,
    const char* index_name);

The call blocks until the SDK operation finishes and uses the existing error-reporting convention. It validates null arguments and string encoding and preserves dataset/session ownership for the call lifetime.

Lance Rust already provides prewarm by logical index name, including multiple segments. The C binding delegates to that existing method. Integration tests must verify compatibility and cache reuse.

Dependency baseline: lance-c 98468344bc9d56aa7ed4a4192e844afa62b34ba1 includes the public C/C++ prewarm binding from lance-c #94 and pins Lance 68c12dfd7efe02f90ad2d7f3a239a7eb64884e57. The Doris dependency update merged into branch-4.1 in Doris #68698 and into master in Doris #68765 on 2026-10-08.

Doris FE/BE

• Extend the existing WARM UP SELECT parser and command with a validated statement-local index-only setting.
• Reuse catalog access and snapshot resolution, including the normal temporary-credential path where applicable. Keep credentials scoped to the normal execution/access path and exclude them from logs and user-visible results.
• Add a small BE RPC handler that opens the pinned dataset through the existing LanceSessionManager and invokes the binding. Return status directly.
• Bound RPC fan-out and concurrent prewarm work per BE using existing execution/resource controls. The synchronous user interface does not require serial execution across every BE.
• Reuse the SDK's bounded windowed IO. Do not implement per-partition RPCs or issue synthetic ANN queries to warm the cache.
• Use normal mixed-version capability/error handling so an old BE cannot silently report success without doing the operation.

Cache scope and expected benefit

The existing LanceSessionManager owns one shared session per BE process. The prewarm handler must use it rather than constructing a temporary private session whose cache disappears after the statement.

Current Doris configuration exposes lance_index_cache_size_bytes and lance_metadata_cache_size_bytes, with current defaults of 10 GiB and 1 GiB per BE. These are process-scoped settings in the existing manager. Verify values against the delivery branch.

A successful statement means all requested SDK calls completed. It does not guarantee permanent or complete residency: entries remain evictable, and a working set larger than the cache cannot stay fully warm. Index files' encoded size is not necessarily their decoded memory footprint. Document capacity planning and retain bounded IO/decode concurrency.

For N target BEs and a working set of about S bytes, full replication needs approximately S on each BE and N * S aggregate capacity. It can also multiply prewarm traffic. Cache on another BE cannot substitute for missing entries on the BE executing a query.

The pinned SDK IVF prewarm implementation already groups adjacent partitions into bounded read windows. Reuse that path to move cold index reads ahead of foreground queries and reduce small remote reads during subsequent queries.

Prewarm still consumes S3 IO itself. It may increase total traffic for rarely queried indexes. It does not automatically warm row-fetch/output columns, data-file filter columns, unindexed rows, or all version-dependent metadata. It does not eliminate distance-computation/Top-K CPU work.

Cache effectiveness and invalidation

Lance scopes index cache entries by dataset URI and index UUID, with additional remapping identity where needed. Reuse the normal query access path and check that prewarm cannot bypass authorization or alias different storage namespaces. Do not flush all immutable segments merely because the dataset version changes.

There is no new cache TTL, pinning guarantee, persistent index cache, automatic refresh, or topology-driven rewarm in this first implementation.

Tests and acceptance

• FE tests: both warmup modes, setting validation, column/index selection, session compute-group resolution, permissions, pinned snapshot/index resolution, and target selection.
• Lance-C/BE tests: null/invalid inputs, missing indexes, multi-segment indexes, shared-session reuse across separately opened dataset handles, proper resource release, and error propagation.
• End-to-end regression: synchronous completion, correct results before/after prewarm, multiple BEs, and an explicit error when any target fails.
• Cache lifecycle tests: shared-session reuse, capacity eviction, BE restart, index replacement, and snapshot visibility/deletes.
• Establish and publish the supported index-type/format and catalog-access matrix. Do not claim untested scalar/FTS or vector format combinations are covered.
• In a controlled environment with enough cache and no eviction, subsequent queries should perform approximately zero remote index-content reads. Measure metadata and row-fetch IO separately. Test multiple vectors and nprobes values rather than replaying only one query.
• Measure prewarm duration/traffic, subsequent S3 GET/Range count and bytes, cold/warm latency, peak memory, and interference with concurrent foreground queries. Exact new per-operation metrics are not a prerequisite; use existing trustworthy metrics or isolated instrumented tests. Do not treat a concurrent global cache-stat delta as task-local IO.
• Add bilingual Doris website documentation for syntax, default execution context, synchronous behavior, errors, cache lifecycle, capacity planning, and benefit boundaries.

Roadmap

Stages are ordered by implementation dependency. Completion requires code, tests, documentation, and verification evidence.

M1 — Upstream Lance-C binding

Upstream implementation: lance-c #94 is merged. The synchronous C/C++ APIs, argument/error handling, multi-segment and shared-session tests, and usage documentation are available upstream. Doris dependency delivery is complete: Doris #68698 merged into branch-4.1, and Doris #68765 merged into master, on 2026-10-08. M1 is complete; M2 and M3 remain open.

• Add the synchronous prewarm-by-index-name C API over the existing Rust SDK method.
• Add argument/error, multi-segment, shared-session reuse, and resource-lifetime tests plus a usage example.
• Merge the Lance-C binding upstream.
• Merge the Doris dependency update without a Doris-only prewarm patch.

The existing Foyer data-file cache patch is retained until lance-c #73 merges; it contains no downstream prewarm implementation.

M2 — Synchronous Doris SQL integration

Implementation PR: Doris #68775, targeting branch-4.1.

• Extend WARM UP SELECT with read_index_only properties, column/index selection, and permission checks.
• Resolve a fixed snapshot and eligible target BEs; implement bounded RPC fan-out and synchronous waiting.
• Invoke the C API through LanceSessionManager and return explicit success/error.
• Add FE/BE unit tests and external regression coverage.

M3 — Verification, documentation, and delivery

• Verify the supported index/catalog matrix, cache lifecycle, capacity behavior, and failed-target handling.
• Benchmark remote index-read reduction and foreground-query impact with reproducible cold/warm conditions.
• Publish bilingual user documentation and merge delivery into branch-4.1.
• Link upstream, Doris, documentation PRs and verification evidence before closing this issue.

Code references

• Lance dataset index API
• Lance IVF prewarm
• Doris shared session manager