feat: Add a non-JVM write path for Lance data sources - #6945

Rebased onto master now that #6943 (the read path) and #6944 (fixed_size_list decoding) have merged. The two commits that carried them are gone; what is left is the write path alone, and it applies cleanly.

What this does

#6943 made a LanceSource readable without Spark. This makes one writable, through the same two addressing modes — a uri, or a namespace_client + table_id resolved through a Lance namespace — so a catalog-addressed dataset is written to the location its namespace resolves rather than to one assembled locally.

_write_data_source gains a LanceSource branch, following the IcebergSource branch already there, which makes the path reachable from DuckDBOfflineStore.offline_write_batch. The isinstance narrowing that the read path open-coded is now a shared _as_lance_source helper used by both, rather than a second copy of the optional-import guard.

+94 lines in duckdb.py, +121 in lance_source.py.

How far the namespace is involved

Measured by recording the calls a write makes on the namespace client:

So a Lance namespace records that a table exists and where it lives, and does not track its versions — the version an append produces is never reported back to it. This is the reason the pin contract below has to be enforced locally: no catalog is going to reject a write on a pin's behalf. There is a test that records these calls so the behaviour is pinned rather than assumed.

Three things settled before Lance is called

A pinned source is refused. A Lance commit always produces a new version, so write_dataset has no version argument at all. A write through a source pinned to version 1 or to a tag would succeed and then be invisible through the very source that performed it. Measured:

tag prod -> version 1
appended; latest ver 2 | tag prod still reads rows: 10   <-- write invisible to pinned source

An absent dataset is created rather than appended to, because Lance has no create-or-append mode:

CREATE on existing: REJECTED -> OSError Dataset already exists

An existing dataset's vector widths are compared against the incoming data. This is the one case Lance itself permits silently:

APPEND dim-mismatch:  REJECTED -> `emb` should have type fixed_size_list:float:8 but type was fixed_size_list:float:16
OVERWRITE 8->32:      fixed_size_list<item: float>[32]  ver 3    <-- accepted, schema silently rewritten
  v1 still readable:  fixed_size_list<item: float>[8]            <-- history now self-inconsistent

So a single overwrite can leave a dataset whose version history has mutually incomparable vectors, with every ANN index built on the old width invalidated. No legitimate schema evolution changes an embedding's dimension, so this is refused rather than carried out.

The guard is narrow on purpose: only vector widths are policed. Other type changes stay Lance's business, because for those an overwrite is a defensible evolution — there is a test asserting an int64 → string overwrite still goes through.

Declared vector_length

Checked at the store entry point, not in the writer callback, which is handed a DataSource and so cannot see the feature view. It reuses _validate_vector_field_lengths from #6909 rather than adding a second implementation. Without it, creating a dataset at a width other than the declared one would succeed and fail only on the next read.

Mode semantics

overwrite without allow_overwrite raises SavedDatasetLocationAlreadyExists, matching the FileSource branch.

persist is not reachable for Lance yet: it needs a SavedDatasetStorage, and _DATA_SOURCE_TO_SAVED_DATASET_STORAGE currently maps only FileSource and SparkSource (IcebergSource has no entry either). I left that for a follow-up because the obvious slot is a problem on its own — _proto_attr_name = "custom_storage" is already claimed by three classes (couchbase, clickhouse, postgres) against a plain-dict registry, so SavedDatasetStorage.from_proto dispatches to whichever was imported last. I raised that as #6946 rather than add a fourth claimant. overwrite is still implemented and tested directly, since it is required for correctness once that lands.

A Spark write path is also deliberately out of scope here. SparkOfflineStore.offline_write_batch consults only file_format and ignores table_format, which I raised as #6955; that needs a decision about which of the two axes is authoritative before a Lance branch there would mean anything.

Verification

20 tests added; 65 pass in test_lance_source.py (45 from #6943 plus these). Catalog-based cases use the dir namespace implementation, which drives the identical namespace_client + table_id code path as a remote catalog, so no server is needed.

Covered: create and append for both addressing modes; version-pin and tag-pin refusal; a refused write leaving the dataset at its original version and row count; the append-invisible-to-an-earlier-pin property that motivates the refusal; overwrite with and without allow_overwrite; vector-width change refused and width-preserving overwrite allowed; a non-vector type change left alone; a width guard ignoring columns absent from the write; declared-width enforcement at offline_write_batch; the namespace call sequence above; and a write-then-read round trip through _read_data_source.

Full sdk/python/tests/unit: 40 failures on master and 53 on this branch, but the 13 names in the difference all reproduce on untouched master when that subset is run in isolation — they are pre-existing order- and environment-dependent cases in test_milvus_online_store, test_chronon_online_store and test_metrics, not regressions. Nothing in the difference touches Lance.

ruff check sdk/python/ and ruff format --check clean.

One caveat worth stating plainly: as Copilot noted on #6943, this test module is skipped entirely by its module-level importorskip, so these tests do not run in CI today. #6958 adds the lance extra and wires it into feast[ci], which is what will actually turn them on. Until it lands, the numbers above are local.