Visitar URL original
fix: Provision online store infra for remote feast apply by RishabhCodezZz · Pull Request #6772 · feast-dev/feast · GitHub
Skip to content

fix: Provision online store infra for remote feast apply - #6772

Open
RishabhCodezZz wants to merge 9 commits into
feast-dev:masterfrom
RishabhCodezZz:fix/6693-remote-online-store-provisioning
Open

RishabhCodezZz wants to merge 9 commits into
feast-dev:masterfrom
RishabhCodezZz:fix/6693-remote-online-store-provisioning

Conversation

@RishabhCodezZz

@RishabhCodezZz RishabhCodezZz commented Aug 21, 2026 •

Copy link
Copy Markdown

What this PR does / why we need it:

feast apply against online_store: type: remote registers feature views in the registry but never creates their backing tables. A subsequent feast materialize fails with HTTP 500 (sqlite3.OperationalError: no such table).

Root cause: RemoteOnlineStore.update() and teardown() were both pass — no provisioning call was ever forwarded to the server. The feature server refreshes the registry via async_refresh() but never calls update_infra(). Since table DDL lives exclusively in each concrete store's update() method, this affects every online store behind type: remote, not just SQLite.

Fix: RemoteOnlineStore.update() and teardown() now call two new feature server endpoints (/update-infra and /teardown-infra) that run the real provisioning against the server's online store.

Objects travel in the request body as base64-encoded protobufs rather than being looked up by name, because FeatureStore.apply() calls update_infra() before registry.commit() — at that point the server can't see them yet. Authorization reuses existing CREATE/DELETE actions (no new permissions).

Changes:

File Change
feast/infra/online_stores/remote.py Implemented update() and teardown(); added encode/decode helpers and HTTP functions
feast/feature_server.py Added POST /update-infra and POST /teardown-infra endpoints with request models
tests/integration/.../test_remote_online_store_provisioning.py New end-to-end regression test covering both endpoints
tests/unit/.../test_remote_online_store.py +7 unit tests (serialization roundtrip, success/error/rejection paths)
docs/reference/feature-servers/python-feature-server.md Documented new endpoints

Why the existing test didn't catch this: test_remote_online_store_read_write applies driver_hourly_stats, which the server already provisioned during its own setup. The table exists before the remote client touches it. The new test applies a name the server has never seen, and asserts against the server's own SQLite file rather than inferring success from a write returning 200.

Notes for reviewers

Two deliberate choices worth calling out, both easy to change if you would rather go the other way:

1. The endpoints call provider.update_infra() / provider.teardown_infra(), not just the online store. PassthroughProvider.update_infra() also invokes batch_engine.update(). For the default engine that is a no-op, but for engines that provision real resources this means a remote feast apply now provisions batch-engine infra server-side too. I think that is right — in remote mode the server is the side that actually holds the infra config — but it is broader than the "online store" framing in the title, so say the word if you would prefer it narrowed to online_store.update().

2. Version skew is now a loud failure instead of a silent one. A client on this version talking to a feature server that predates these endpoints gets a 404, which surfaces as a RuntimeError out of feast apply. Previously the call silently did nothing and the problem only showed up later as no such table during materialization. Failing at apply time seems strictly better, but it is a behaviour change for mixed-version deployments and probably deserves a release note.

Which issue(s) this PR fixes:

Fixes #6693

Checks

  • I've made sure the tests are passing.
  • My commits are signed off (git commit -s)
  • My PR title follows conventional commits format

Testing Strategy

  • Unit tests
  • Integration tests
  • Manual tests
  • Testing is not required for this change

Misc

Design choice approved by @yuan1j in #6693 — Option A (client-driven, synchronous provisioning via new endpoints).

The first two commits are ordered red-then-green: the failing test lands first, the fix second, so the git history itself proves the test catches the bug. The third commit addresses review feedback — entity-level authorization on both endpoints, caching the lazily-built type map, and extending the integration test to assert the table is present after apply and gone after teardown, so both new endpoints are covered end to end.

Applying a new feature view through a client whose online store is
`type: remote` leaves it unwritable: the feature view reaches the registry,
but no table is provisioned server-side, so the write that `feast materialize`
performs returns HTTP 500 with `no such table`.

The existing test_remote_online_store_read_write does apply a feature view
through a remote client and write to it, but it applies `driver_hourly_stats`
-- a name the server already provisioned during its own `feast apply` at
setup. The table exists before the client touches it, which masks the bug.
This test applies a feature view the server has never seen.

The assertion is on the user-visible contract (a feature view applied through
a remote online store must be writable) rather than on any particular
mechanism, so it holds regardless of how the fix is implemented.

Refs feast-dev#6693

Signed-off-by: Rishabh <158596025+RishabhCodezZz@users.noreply.github.com>
`feast apply` against `online_store: type: remote` registered feature views
but never created their tables, so `feast materialize` failed with HTTP 500
and `no such table`. `RemoteOnlineStore.update()` and `.teardown()` were both
`pass`, and the feature server never calls `update_infra()` on its own -- it
only refreshes the registry. Since table DDL lives exclusively in the concrete
online store's `update()`, this affected any online store behind `type: remote`,
not just SQLite.

RemoteOnlineStore.update()/teardown() now call two new feature server
endpoints, /update-infra and /teardown-infra, which run update_infra() and
teardown_infra() against the server's real online store.

The objects travel in the request body as base64-encoded protos rather than
being looked up server-side by name, because FeatureStore.apply() calls
update_infra() *before* registry.commit() -- at that point the server cannot
see them yet, and with a file registry they are not even on disk.

Authorization reuses the existing CREATE and DELETE actions, so no new
permission surface is introduced.

Fixes feast-dev#6693

Signed-off-by: Rishabh <158596025+RishabhCodezZz@users.noreply.github.com>
@RishabhCodezZz
RishabhCodezZz requested a review from a team as a code owner August 21, 2026 16:25
Copilot AI lite review requested due to automatic review settings August 21, 2026 16:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a remote-mode infrastructure provisioning gap in Feast: when online_store.type: remote, a client-side feast apply previously registered objects but did not provision/drop the backing online-store tables on the feature server, causing later writes/materialization to fail.

Changes:

  • Implement RemoteOnlineStore.update() / teardown() to call feature server endpoints to provision and tear down online-store infrastructure.
  • Add POST /update-infra and POST /teardown-infra endpoints to the Python feature server to run provider infra operations server-side.
  • Add unit + integration regression coverage and document the new endpoints.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
sdk/python/feast/infra/online_stores/remote.py Adds infra object (de)serialization and forwards update/teardown to feature server endpoints.
sdk/python/feast/feature_server.py Introduces /update-infra and /teardown-infra endpoints that invoke provider infra operations.
sdk/python/tests/unit/infra/online_store/test_remote_online_store.py Adds unit tests for serialization and update/teardown request/response handling.
sdk/python/tests/integration/online_store/test_remote_online_store_provisioning.py Adds end-to-end regression test ensuring remote apply provisions a previously-unknown FV table.
docs/reference/feature-servers/python-feature-server.md Documents the new infra endpoints in the feature server API table.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +99 to +104
type_name = type(obj).__name__
if type_name not in _infra_object_types():
raise ValueError(
f"Cannot send {type_name} to the remote online store; "
f"expected one of {sorted(_infra_object_types())}"
)
Comment on lines +977 to +980
for table in tables_to_keep:
assert_permissions(resource=table, actions=[AuthzedAction.CREATE])
for table in tables_to_delete:
assert_permissions(resource=table, actions=[AuthzedAction.DELETE])
Comment on lines +119 to +122
proc.kill()
proc.wait(timeout=30)
log_file.close()

Comment on lines +1002 to +1004
for table in tables:
assert_permissions(resource=table, actions=[AuthzedAction.DELETE])

RishabhCodezZz and others added 3 commits August 22, 2026 22:16
Review follow-ups on the remote online store provisioning endpoints.

/update-infra and /teardown-infra asserted permissions on the feature views
in the request body but not on the entities, so a caller could provision or
drop entity infrastructure without an authorization check. Both endpoints now
apply the same CREATE and DELETE actions to entities that they already apply
to tables.

_infra_object_types() re-ran its five lazy imports on every call, and
encode_infra_object() invoked it twice per object across four lists. It is
now cached and read once per call.

The integration test proved that apply() provisions a table but never
exercised the teardown path, and both of its fixtures leaked their temp
directories. It now asserts against the server's own SQLite file on both
sides of the lifecycle -- the table exists after apply and is gone after
teardown, which covers /update-infra and /teardown-infra end to end -- and
removes its temp directories on the way out.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Rishabh <158596025+RishabhCodezZz@users.noreply.github.com>
@yuan1j

yuan1j commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Friendly ping @feast-dev/reviewers-and-approvers — the Checks tab still shows "12 workflows awaiting approval", so CI has never run on this PR.

Could someone with permissions approve the workflow run and/or request a reviewer? The Copilot comments were addressed in 70459ce back on 08-22. I'm the reporter of #6693 and happy to help with anything that speeds this up.


def _free_port() -> int:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(("", 0))
@codecov-commenter

codecov-commenter commented Sep 11, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 67.39130% with 30 lines in your changes missing coverage. Please review.
✅ Project coverage is 47.13%. Comparing base (8ab92e8) to head (f24aadf).
⚠️ Report is 8 commits behind head on master.

Files with missing lines Patch % Lines
sdk/python/feast/feature_server.py 37.14% 22 Missing ⚠️
sdk/python/feast/infra/online_stores/remote.py 85.96% 8 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #6772      +/-   ##
==========================================
+ Coverage   47.08%   47.13%   +0.05%     
==========================================
  Files         419      419              
  Lines       51878    51971      +93     
  Branches     7525     7538      +13     
==========================================
+ Hits        24429    24499      +70     
- Misses      25700    25723      +23     
  Partials     1749     1749              
Flag Coverage Δ
go-feature-server 30.58% <ø> (ø)
python-unit 48.44% <67.39%> (+0.05%) ⬆️
Files with missing lines Coverage Δ
sdk/python/feast/infra/online_stores/remote.py 67.09% <85.96%> (+3.23%) ⬆️
sdk/python/feast/feature_server.py 61.66% <37.14%> (-1.23%) ⬇️

... and 4 files with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8ab92e8...f24aadf. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@RishabhCodezZz

Copy link
Copy Markdown
Author

The 6 failing test cases are not related to this branch.

lint-python, smoke-test-python, and unit-test-python on Linux (3.10/3.11/3.12) are all failing at the same point: the Install dependencies step, due to a hash mismatch for torchvision==0.28.0+cpu. This is a master CI issue, not something introduced by this diff — this branch doesn't touch pyproject.toml or anything under sdk/python/requirements/.

#6825 already fixes this by recompiling the lockfiles with --universal, which records the correct hashes for the CPU builds. I also checked the CI for that PR, and every job that's red here is green there.

The macOS 3.12 failure is a separate issue. That job ran 2,638 tests before timing out on test_module_level_fips_sets_env_before_pyarrow_import, which is a subprocess test unrelated to this PR. Dependency installation completed successfully on macOS (the torch-backend flag is Linux-only), so this isn't the same root cause. It looks like a flake.

I'll pull master back in once #6825 merges, and these should clear up.

RishabhCodezZz and others added 2 commits September 15, 2026 08:49
Security fix found in review of the /update-infra and /teardown-infra
endpoints added in bd367e4.

`decode_infra_object()` called `from_proto()` on client-supplied
protobuf bytes without `skip_udf=True`. For StreamFeatureView, that
means `dill.loads()` runs on the embedded UDF body; for
OnDemandFeatureView with a pandas transformation, `resolve_udf()`
runs `exec()` on the UDF source text. Both are documented elsewhere
in this codebase as safe only for trusted registry content --
registry_server.py passes `skip_udf=True` in every place it decodes
a client-submitted proto for exactly this reason. These two new
endpoints decode a raw HTTP request body from an authenticated but
untrusted client, so neither unpickling nor exec'ing it is safe.
Confirmed exploitable: a crafted StreamFeatureView proto with a
poisoned pickle payload, and a crafted OnDemandFeatureView proto
with a malicious body_text, both run arbitrary code during decode,
before authorization is checked and independent of whether the
proto is otherwise valid.

`_infra_object_types()` now records which of the five object kinds
carry a UDF, so `decode_infra_object()` cannot decode one without
passing `skip_udf=True`. Provisioning only ever needs schema and
metadata, never the transformation itself, so this has no functional
cost. Two regression tests prove both vectors are closed -- each
fails against the pre-fix code and passes against the fix.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Rishabh <158596025+RishabhCodezZz@users.noreply.github.com>
`/update-infra` and `/teardown-infra` have authorized entities
alongside feature views since 70459ce, but this table was written
in the commit before that (bd367e4) and was never updated to match.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Rishabh <158596025+RishabhCodezZz@users.noreply.github.com>
@RishabhCodezZz

Copy link
Copy Markdown
Author

Two additional commits have been pushed to this PR, both self-reviews prior to anyone from the maintainer team reviewing it:

f24aadf1 — decode_infra_object() was calling from_proto() on the raw client-supplied bytes without skip_udf=True. For StreamFeatureView/OnDemandFeatureView/FeatureView, that means the embedded UDF gets unpickled or exec'd during decode. registry_server.py always passes skip_udf=True when decoding client-submitted protos for exactly this reason; these two new endpoints weren't. Fixed, with two regression tests that build a malicious payload and confirm no code executes — verified against the pre-fix code first, so they're not tautological.

51813b8b — the permissions table in the docs still listed FeatureView as the only authorized resource type on /update-infra//teardown-infra. It was written before entity-level authorization was added and never got updated. Now says FeatureView,Entity, matching what the code actually checks.

Both are patches on top of what's already here — nothing in the original design or the endpoints themselves was altered.

@RishabhCodezZz
RishabhCodezZz requested a lite review from Copilot September 15, 2026 11:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

'feast apply' does not create online store tables in remote mode

5 participants