Visitar URL original
feat: Add a non-JVM write path for Lance data sources by haoxu0 · Pull Request #6945 · feast-dev/feast · GitHub
Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix: Correct what a Lance namespace does during a write
The docstring claimed a catalog-addressed write is "committed through its
namespace". That is only true of a create. Instrumenting the namespace
client shows how far it is actually involved:

  create -> declare_table
  append -> describe_table only
  open   -> describe_table, namespace_id

So a namespace records that a table exists and where it lives, and does
not track its versions: the version an append produces is never reported
back to it. Worth stating precisely, because it is the reason the pin
contract has to be enforced in `assert_writable` rather than left to the
catalog -- no catalog is going to reject a write on a pin's behalf.

Adds a test that records the namespace calls, so the claim is pinned by a
test rather than asserted in prose.

Signed-off-by: hao-xu5 <hxu44@apple.com>
  • Loading branch information
haoxu0 authored and ntkathole committed Oct 9, 2026
commit f96470b9ae413b3a1c69e1c2aa279c742432b30c
12 changes: 10 additions & 2 deletions sdk/python/feast/infra/offline_stores/duckdb.py
Original file line number Diff line number Diff line change
Expand Up @@ -257,8 +257,16 @@ def _write_lance_data_source(

Mirrors ``_read_lance_source``: the source resolves either a uri or a
``namespace_client`` + ``table_id``, and the same resolution is reused for
the write so that a catalog-addressed dataset is committed through its
namespace rather than behind its back.
the write, so a catalog-addressed dataset is written to the location its
namespace resolves rather than to one assembled here.

How far the namespace is involved depends on the operation. Creating a table
declares it through the namespace; appending only describes the table to
resolve its location, and the version the append produces is not reported
back. A Lance namespace therefore records that a table exists and where it
lives, and does not track its versions. That is why a pin is a Lance-level
concern that ``assert_writable`` has to enforce here: no catalog is going to
reject the write on the pin's behalf.

Three things are settled before Lance is called. A pinned source is refused,
because a commit produces a new version that the pin would not read. An
Expand Down
47 changes: 47 additions & 0 deletions sdk/python/tests/unit/infra/data_sources/test_lance_source.py
Original file line number Diff line number Diff line change
Expand Up @@ -1011,3 +1011,50 @@ def test_offline_write_batch_appends_a_declared_width_batch(tmp_path):

assert lance.dataset(uri).count_rows() == 4
assert lance.dataset(uri).version == 2


def test_a_namespace_resolves_a_write_but_does_not_track_its_versions(dir_namespace):
"""Pins how far a Lance namespace is involved in a write.

Creating a table declares it through the namespace. Appending only describes
the table, to resolve its location; the version the append produces is not
reported back. So a namespace records that a table exists and where it
lives, and is not a version coordinator -- which is why `assert_writable`
has to enforce the pin contract locally.
"""
client, properties = dir_namespace
calls: list[str] = []

class RecordingNamespace:
def __init__(self, inner):
self._inner = inner

def __getattr__(self, name):
attribute = getattr(self._inner, name)
if not callable(attribute):
return attribute

def recorded(*args, **kwargs):
calls.append(name)
return attribute(*args, **kwargs)

return recorded

source = LanceSource(
table="tracked",
namespace_impl="dir",
namespace_properties=properties,
timestamp_field="event_timestamp",
)
spy = RecordingNamespace(client)
source.get_namespace_client = lambda: spy # type: ignore[method-assign]

_write_lance_data_source(_ibis_table(_driver_stats(n_rows=3)), source)
assert "declare_table" in calls

calls.clear()
_write_lance_data_source(_ibis_table(_driver_stats(n_rows=2, base_hour=12)), source)
assert "declare_table" not in calls
assert "describe_table" in calls

assert source.to_arrow().num_rows == 5