Repository navigation
Ingestion should fail immediately when there are no valid stores #12
Description
Activity
It's a band-aid because if someone alters the DB the job will fail since the checks are only at feature registration time.
And nice! If it works on remote flink we're golden :)
This should stop those self imposed DoS events every 10 minutes too.
We can close this now?
Yep. Thanks for the hard work :)
- added a commit that references this issue
on Jul 29, 2020 - added a commit that references this issue
on Apr 15, 2025 - added a commit that references this issue
on Apr 29, 2025 - added 2 commits that reference this issue
on Jun 7, 2025
Metadata
Metadata
Assignees
Labels
No labels
Expected Behavior
Starting a job with invalid stores (e.g. using redis as a warehouse store) should not be allowed, and should fail quickly - ideally at the graph building step of ingestion.
Current Behavior
Starting a job with invalid stores will successfully send the job to the runner, which will run to completion (or indefinitely, in the case of streaming jobs). The errors will be logged, but the pipeline will run with no problems.
Steps to reproduce
Possible Solution
PR #11 is a band-aid solution to this problem: it checks the store types at registration, ensuring that a feature is unable to use a serving store for warehousing, but ideally ingestion error out properly during graph building.