feat: Add a non-JVM read path for Lance data sources - #6943

feast-dev#6943 made a `LanceSource` readable without Spark. This makes one writable,
through the same two addressing modes: a uri, or a `namespace_client` plus
`table_id` resolved through a Lance namespace, so a catalog-addressed
dataset is committed through its catalog rather than behind its back.

`_write_data_source` gains a `LanceSource` branch, following the
`IcebergSource` branch already there, which makes the path reachable from
`DuckDBOfflineStore.offline_write_batch`. The `isinstance` narrowing the
read path open-coded is now a shared `_as_lance_source` helper used by
both, rather than a second copy of the optional-import guard.

Three things are settled before Lance is called.

A pinned source is refused. A Lance commit always produces a new version,
so `write_dataset` has no `version` argument at all; a write through a
source pinned to version 1 or to a tag would succeed and then be invisible
through the very source that performed it. Measured: after appending to a
dataset tagged `prod` at version 1, the unpinned source reads 5 rows and
the pinned source still reads 3.

An absent dataset is created rather than appended to, because Lance has no
create-or-append mode and rejects `create` on an existing dataset.

An existing dataset's vector widths are compared against the incoming
data. Lance rejects an `append` whose schema disagrees, but it accepts an
`overwrite` that replaces a 8-wide embedding column with a 32-wide one,
silently rewriting the declared shape and leaving a version history whose
vectors are not mutually comparable, with every index built on the old
width invalidated. No legitimate schema evolution changes an embedding's
dimension, so this is refused. The guard is narrow on purpose: only vector
widths are policed, and other type changes remain Lance's business.

The declared `vector_length` is checked at the store entry point rather
than in the writer callback, which is handed a `DataSource` and so cannot
see the feature view. It reuses `_validate_vector_field_lengths` from
 feast-dev#6909 rather than adding a second implementation. Without it, creating a
dataset at a width other than the declared one would succeed and fail only
on the next read.

19 tests added, all using the `dir` namespace implementation for the
catalog-based cases so no server is needed.

Signed-off-by: hao-xu5 <hxu44@apple.com>

feast-dev#6943 made a `LanceSource` readable without Spark. This makes one writable,
through the same two addressing modes: a uri, or a `namespace_client` plus
`table_id` resolved through a Lance namespace, so a catalog-addressed
dataset is committed through its catalog rather than behind its back.

`_write_data_source` gains a `LanceSource` branch, following the
`IcebergSource` branch already there, which makes the path reachable from
`DuckDBOfflineStore.offline_write_batch`. The `isinstance` narrowing the
read path open-coded is now a shared `_as_lance_source` helper used by
both, rather than a second copy of the optional-import guard.

Three things are settled before Lance is called.

A pinned source is refused. A Lance commit always produces a new version,
so `write_dataset` has no `version` argument at all; a write through a
source pinned to version 1 or to a tag would succeed and then be invisible
through the very source that performed it. Measured: after appending to a
dataset tagged `prod` at version 1, the unpinned source reads 5 rows and
the pinned source still reads 3.

An absent dataset is created rather than appended to, because Lance has no
create-or-append mode and rejects `create` on an existing dataset.

An existing dataset's vector widths are compared against the incoming
data. Lance rejects an `append` whose schema disagrees, but it accepts an
`overwrite` that replaces a 8-wide embedding column with a 32-wide one,
silently rewriting the declared shape and leaving a version history whose
vectors are not mutually comparable, with every index built on the old
width invalidated. No legitimate schema evolution changes an embedding's
dimension, so this is refused. The guard is narrow on purpose: only vector
widths are policed, and other type changes remain Lance's business.

The declared `vector_length` is checked at the store entry point rather
than in the writer callback, which is handed a `DataSource` and so cannot
see the feature view. It reuses `_validate_vector_field_lengths` from
 feast-dev#6909 rather than adding a second implementation. Without it, creating a
dataset at a width other than the declared one would succeed and fail only
on the next read.

19 tests added, all using the `dir` namespace implementation for the
catalog-based cases so no server is needed.

Signed-off-by: hao-xu5 <hxu44@apple.com>

* feat: Add a non-JVM write path for Lance data sources

#6943 made a `LanceSource` readable without Spark. This makes one writable,
through the same two addressing modes: a uri, or a `namespace_client` plus
`table_id` resolved through a Lance namespace, so a catalog-addressed
dataset is committed through its catalog rather than behind its back.

`_write_data_source` gains a `LanceSource` branch, following the
`IcebergSource` branch already there, which makes the path reachable from
`DuckDBOfflineStore.offline_write_batch`. The `isinstance` narrowing the
read path open-coded is now a shared `_as_lance_source` helper used by
both, rather than a second copy of the optional-import guard.

Three things are settled before Lance is called.

A pinned source is refused. A Lance commit always produces a new version,
so `write_dataset` has no `version` argument at all; a write through a
source pinned to version 1 or to a tag would succeed and then be invisible
through the very source that performed it. Measured: after appending to a
dataset tagged `prod` at version 1, the unpinned source reads 5 rows and
the pinned source still reads 3.

An absent dataset is created rather than appended to, because Lance has no
create-or-append mode and rejects `create` on an existing dataset.

An existing dataset's vector widths are compared against the incoming
data. Lance rejects an `append` whose schema disagrees, but it accepts an
`overwrite` that replaces a 8-wide embedding column with a 32-wide one,
silently rewriting the declared shape and leaving a version history whose
vectors are not mutually comparable, with every index built on the old
width invalidated. No legitimate schema evolution changes an embedding's
dimension, so this is refused. The guard is narrow on purpose: only vector
widths are policed, and other type changes remain Lance's business.

The declared `vector_length` is checked at the store entry point rather
than in the writer callback, which is handed a `DataSource` and so cannot
see the feature view. It reuses `_validate_vector_field_lengths` from
 #6909 rather than adding a second implementation. Without it, creating a
dataset at a width other than the declared one would succeed and fail only
on the next read.

19 tests added, all using the `dir` namespace implementation for the
catalog-based cases so no server is needed.

Signed-off-by: hao-xu5 <hxu44@apple.com>

* fix: Correct what a Lance namespace does during a write

The docstring claimed a catalog-addressed write is "committed through its
namespace". That is only true of a create. Instrumenting the namespace
client shows how far it is actually involved:

  create -> declare_table
  append -> describe_table only
  open   -> describe_table, namespace_id

So a namespace records that a table exists and where it lives, and does
not track its versions: the version an append produces is never reported
back to it. Worth stating precisely, because it is the reason the pin
contract has to be enforced in `assert_writable` rather than left to the
catalog -- no catalog is going to reject a write on a pin's behalf.

Adds a test that records the namespace calls, so the claim is pinned by a
test rather than asserted in prose.

Signed-off-by: hao-xu5 <hxu44@apple.com>

* fix: Accept an optional source in _as_lance_source

offline_write_batch passes FeatureView.batch_source, which is optional, so
the narrow DataSource annotation made mypy reject the call. The body already
handled None, since isinstance(None, LanceSource) is False, so only the
signature changes.

Signed-off-by: hao-xu5 <hxu44@apple.com>

---------

Signed-off-by: hao-xu5 <hxu44@apple.com>