Visitar URL original
fix: Decode fixed_size_list as the matching Feast list value type by haoxu0 · Pull Request #6944 · feast-dev/feast · GitHub
Skip to content

fix: Decode fixed_size_list as the matching Feast list value type - #6944

Merged
haoxu0 merged 2 commits into
feast-dev:masterfrom
haoxu0:fix/fixed-size-list-value-type
Oct 6, 2026
Merged

haoxu0 merged 2 commits into
feast-dev:masterfrom
haoxu0:fix/fixed-size-list-value-type

Conversation

@haoxu0

@haoxu0 haoxu0 commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator

What this does

pa_to_feast_value_type had no handling for fixed_size_list, so decoding an Arrow vector column raised a KeyError:

>>> pa_to_feast_value_type("fixed_size_list<item: float>[8]")
KeyError: 'fixed_size_list<item: float>[8]'

>>> pa_to_feast_value_type("list<item: float>")   # variable-width worked
ValueType.FLOAT_LIST

Only the fixed-width spelling was affected. "fixed_size_list" not in inspect.getsource(feast.type_map) held before this change.

Why it matters

fixed_size_list is how Arrow-native stores spell an embedding column, and it is the type _validate_vector_field_lengths reads a width from — the validator added in #6909. So schema inference failed on exactly the columns that vector support is meant to describe.

Any source whose schema inference routes through pa_to_feast_value_type is affected, not one particular store. It surfaces through get_pyarrow_schema_from_batch_source, so offline_write_batch and feast apply schema inference both hit it.

Approach

A Feast value type carries no width — Field.vector_length carries it — so the two spellings must agree. The fixed-width spelling is rewritten into the variable-width one and then flows through the existing list handling unchanged, which keeps that logic single-sourced rather than duplicating the element-type table.

Nesting is preserved: fixed_size_list<item: list<item: float>>[8] still decodes to VALUE_LIST, matching list<item: list<item: float>>.

Verification

spelling decodes to
fixed_size_list<item: float>[8] FLOAT_LIST
fixed_size_list<item: double>[1536] DOUBLE_LIST
fixed_size_list<item: int64>[4] INT64_LIST
fixed_size_list<item: int32>[4] INT32_LIST
fixed_size_list<item: string>[2] STRING_LIST
fixed_size_list<item: bool>[2] BOOL_LIST
fixed_size_list<item: list<item: float>>[8] VALUE_LIST

Each is asserted equal to its variable-width counterpart, and one test pins the parser against the exact string PyArrow emits (str(pyarrow.list_(pyarrow.float32(), 8))) so the format cannot drift away unnoticed.

5 tests added. sdk/python/tests/unit/test_type_map.py: 24 failures before the change and the same 24 after, by name — all pre-existing TestSparkNativeTypeValidation cases that need pyspark. 247 passed.

ruff check and ruff format --check clean.

`pa_to_feast_value_type` had no handling for `fixed_size_list`, so decoding
an Arrow vector column raised `KeyError: 'fixed_size_list<item: float>[8]'`.
The variable-width spelling `list<item: float>` already decoded to
`FLOAT_LIST`, so only the fixed-width spelling was affected.

That spelling is how Arrow-native stores represent an embedding, and it is
the type `_validate_vector_field_lengths` reads a vector's width from, so
schema inference failed on exactly the columns the vector support added in
 feast-dev#6909 is meant to describe.

A Feast value type carries no width -- `Field.vector_length` carries it --
so the fixed-width spelling is rewritten into the variable-width one and
decodes identically, leaving the list handling single-sourced. Nesting is
preserved, so a nested fixed-size list still decodes to `VALUE_LIST`.

Signed-off-by: hao-xu5 <hxu44@apple.com>
@haoxu0
haoxu0 requested a review from a team as a code owner October 5, 2026 20:26
@codecov-commenter

codecov-commenter commented Oct 5, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 83.33333% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 49.06%. Comparing base (acadb17) to head (49c4f63).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
sdk/python/feast/type_map.py 83.33% 0 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #6944      +/-   ##
==========================================
+ Coverage   49.04%   49.06%   +0.01%     
==========================================
  Files         433      433              
  Lines       54308    54314       +6     
  Branches     7910     7912       +2     
==========================================
+ Hits        26637    26647      +10     
+ Misses      25790    25788       -2     
+ Partials     1881     1879       -2     
Flag Coverage Δ
go-feature-server 30.58% <ø> (ø)
python-unit 50.45% <83.33%> (+0.01%) ⬆️
Files with missing lines Coverage Δ
sdk/python/feast/type_map.py 63.29% <83.33%> (+0.63%) ⬆️

... and 1 file with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update acadb17...49c4f63. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@haoxu0
haoxu0 merged commit 8cfe891 into feast-dev:master Oct 6, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants