feat: Add LanceFormat table format with version/tag pinning - #6925

Adds LanceSource and teaches the DuckDB offline store to read it, completing the
read half of feast-dev#6899. LanceFormat landed in feast-dev#6925 as a format descriptor; nothing
read Lance until now.

Lance already worked through SparkSource, which drives its reader generically
from table_format.format_type.value and table_format.properties. What was
missing is a path that needs no JVM, which is also the real test of whether the
DataSource abstraction is engine-agnostic rather than Spark-agnostic in name
only.

Placement: a new source read by the existing DuckDB store, rather than a Lance
offline store or an extension of FileSource.

FileSource is the wrong host. Its format axis is already taken by file_format,
so adding table_format would give one source two overlapping format axes. It is
also read by two stores with incompatible contracts: duckdb._read_data_source
dispatches on type, while dask._read_datasource has no dispatch seam and reads
file_options.uri unconditionally as Parquet, and asserts isinstance(...,
FileSource) in three places. Decisively, Lance's catalog addressing has no path
to put in FileSource.path, so the catalog-based layer would not fit the class
even if the path-based one did.

A Lance offline store would be the wrong 120 lines. duckdb.py is a binding that
injects reader and writer callbacks into the engine in ibis.py, so a Lance store
would be a near-copy of it plus a repo_config entry, and would force a choice
between Lance and Parquet instead of mixing them in one feature service.

Reading a source as ibis.memtable(arrow_table) in the DuckDB store already has
two precedents, IcebergSource and MlflowDatasetSource. Following them leaves
ibis.py untouched, so the point-in-time join, TTL handling, field mapping and
ODFVs work unchanged, and no edit to repo_config.py or data_source.py is needed
because CUSTOM_SOURCE plus data_source_class_type is self-describing.

Both addressing modes work: a uri, and catalog/namespace/table through
namespace_client and table_id.

Pin semantics follow what was argued on feast-dev#5782 and feast-dev#6925: a pin selects data,
never shape. get_table_column_names_and_types reads the pinned schema so
feast apply infers what reads will actually see; a pre-flight check fails with a
message naming the pin when a pinned version cannot satisfy the declared schema;
and vector widths go through _validate_vector_field_lengths from feast-dev#6909 rather
than a second validator.

Tests use the dir namespace implementation, which exercises the same
namespace_client and table_id code path as a remote catalog with no server
required. 43 tests, including a demonstration that a tag pin returns earlier
data after the dataset has been overwritten for the same entity and timestamp.

Read-only for now: _write_data_source is untouched, so a LanceSource is not yet
a persist target and there is no SavedDatasetLanceStorage.

Signed-off-by: hao-xu5 <hxu44@apple.com>

* feat: Add a non-JVM read path for Lance data sources

Adds LanceSource and teaches the DuckDB offline store to read it, completing the
read half of #6899. LanceFormat landed in #6925 as a format descriptor; nothing
read Lance until now.

Lance already worked through SparkSource, which drives its reader generically
from table_format.format_type.value and table_format.properties. What was
missing is a path that needs no JVM, which is also the real test of whether the
DataSource abstraction is engine-agnostic rather than Spark-agnostic in name
only.

Placement: a new source read by the existing DuckDB store, rather than a Lance
offline store or an extension of FileSource.

FileSource is the wrong host. Its format axis is already taken by file_format,
so adding table_format would give one source two overlapping format axes. It is
also read by two stores with incompatible contracts: duckdb._read_data_source
dispatches on type, while dask._read_datasource has no dispatch seam and reads
file_options.uri unconditionally as Parquet, and asserts isinstance(...,
FileSource) in three places. Decisively, Lance's catalog addressing has no path
to put in FileSource.path, so the catalog-based layer would not fit the class
even if the path-based one did.

A Lance offline store would be the wrong 120 lines. duckdb.py is a binding that
injects reader and writer callbacks into the engine in ibis.py, so a Lance store
would be a near-copy of it plus a repo_config entry, and would force a choice
between Lance and Parquet instead of mixing them in one feature service.

Reading a source as ibis.memtable(arrow_table) in the DuckDB store already has
two precedents, IcebergSource and MlflowDatasetSource. Following them leaves
ibis.py untouched, so the point-in-time join, TTL handling, field mapping and
ODFVs work unchanged, and no edit to repo_config.py or data_source.py is needed
because CUSTOM_SOURCE plus data_source_class_type is self-describing.

Both addressing modes work: a uri, and catalog/namespace/table through
namespace_client and table_id.

Pin semantics follow what was argued on #5782 and #6925: a pin selects data,
never shape. get_table_column_names_and_types reads the pinned schema so
feast apply infers what reads will actually see; a pre-flight check fails with a
message naming the pin when a pinned version cannot satisfy the declared schema;
and vector widths go through _validate_vector_field_lengths from #6909 rather
than a second validator.

Tests use the dir namespace implementation, which exercises the same
namespace_client and table_id code path as a remote catalog with no server
required. 43 tests, including a demonstration that a tag pin returns earlier
data after the dataset has been overwritten for the same entity and timestamp.

Read-only for now: _write_data_source is untouched, so a LanceSource is not yet
a persist target and there is no SavedDatasetLanceStorage.

Signed-off-by: hao-xu5 <hxu44@apple.com>

* fix: Address Lance read review feedback

Signed-off-by: HaoXuAI <sduxuhao@gmail.com>

---------

Signed-off-by: hao-xu5 <hxu44@apple.com>
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>