Visitar URL original
Add upstream FeatureView lineage support for PushSource · Issue #6839 · feast-dev/feast · GitHub
Skip to content

Add upstream FeatureView lineage support for PushSource #6839

Description

@mao-liu

Is your feature request related to a problem? Please describe.

In many real-world production ML architectures, derived features are computed outside of Feast by external stream or batch computation engines. These external pipelines read upstream features from existing Feast FeatureViews and push the transformed features into Feast via a PushSource backed by an offline batch store.

Currently, Feast's static registry lineage only tracks:

PushSource --> Target FeatureView

Because PushSource does not support declaring the upstream FeatureViews it was computed from, the relationship between the upstream feature views and the push source is lost in both the Feast UI Lineage graph and the registry metadata.

Describe the solution you'd like

We would like PushSource to support declaring upstream FeatureView dependencies via a source_views (or upstream_feature_views) parameter:

# 1. Upstream feature views
user_tx_fv = FeatureView(name="user_transaction_stats", ...)
user_credit_fv = FeatureView(name="user_credit_profile", ...)
# 2. PushSource declaring upstream views and offline batch source
risk_push_source = PushSource(
    name="risk_calc_pipeline",
    batch_source=FileSource(path="data/risk_offline.parquet"),
    source_views=[user_tx_fv, user_credit_fv],  # <-- Upstream dependencies
    description="External Spark job computing risk scores from transaction & credit features",
)
# 3. Target feature view receiving the pushed features
user_risk_fv = FeatureView(
    name="user_risk_target_fv",
    entities=[user_entity],
    ttl=timedelta(days=30),
    source=risk_push_source,
    schema=[Field(name="risk_score", dtype=Float32)],
)

With this metadata, Feast can render the complete directed graph in the Feast UI Lineage view:

[ user_transaction_stats ] (FeatureView) ──┐
                                           ├──→ [ risk_calc_pipeline ] (PushSource) ──→ [ user_risk_target_fv ] (FeatureView)
[ user_credit_profile ]    (FeatureView) ──┘

Proposed Implementation Details

  1. Protobuf (protos/feast/core/DataSource.proto): Add upstream_feature_views to PushOptions:
message PushOptions {
  reserved 1;
  repeated string upstream_feature_views = 2;
}
  1. Python SDK (sdk/python/feast/data_source.py): Update PushSource constructor, to_proto(), and from_proto() to accept and serialize source_views: Optional[List[Union[BaseFeatureView, str]]].
  2. Registry Lineage (sdk/python/feast/lineage/registry_lineage.py): Update RegistryLineageGenerator._parse_direct_relationships() to emit EntityRelation(source=FeatureView, target=DataSource) for each upstream feature view defined on a PushSource.
  3. UI Lineage Parser (ui/src/parsers/parseEntityRelationships.ts): Extract upstreamFeatureViews from dataSources to draw the corresponding edges in React Flow.
  4. OpenLineage Emitter (sdk/python/feast/openlineage/mappers.py): Ensure emit_apply() creates the corresponding input dataset connections for OpenLineage producers.

Describe alternatives you've considered

  1. Using OnDemandFeatureView (ODFV): ODFV handles on-the-fly transformations at request/serving time, but is not intended for pre-computed, stored, push-ingested features requiring offline batch storage.
  2. Adding metadata in tags: Storing tags={"depends_on": "user_tx_fv,user_credit_fv"} records metadata, but is non-standard and does not render visual lineage relationships in the Feast UI graph.
  3. Emitting manual OpenLineage events via CI/CD: Works if an OpenLineage backend is configured, but does not solve native Feast static registry lineage or out-of-the-box UI visualization.

Additional context
I am happy to contribute this change and submit a PR covering the proto update, Python SDK, registry lineage generator, and UI relationship parser if the community agrees with this direction!

Activity

  1. ntkathole commented on Sep 15, 2026

    @ntkathole
    Member

    @mao-liu Thanks for raising it, it's a valid feature request and would be a good contribution to accept. I can help with reviews.

  2. mao-liu commented on Sep 17, 2026

    @mao-liu
    Author

    Thanks @ntkathole - letting you know that I have began working on this and testing on my fork.

    There are a couple of pre-existing issues on master that I've identified along the way during testing:

    1. Feast registry lineage doesn't currently render stream sources (affects Push/Kafka/Kinesis sources)
    • only batch sources are currently rendered
    • rendering nodes and edges for stream sources is easy enough to add, and I will include it on my PR
    1. Feast serve_lineage UI currently doesn't correctly display relationships between data sources and feature views
    • Currently, feast apply emits a single openlineage job event with all sources and features in a single event
    • OpenLineageProcessor then computes edges as the cartesian product of all sources and views in the same job - leading to all data sources in a feature repo connected to all feature views
    • Is this the intended behaviour? Would you like me to open a separate issue for the community to track this behaviour?

    We don't have any requirements to use OpenLineage... but wanted to raise that its lineage view might not be very visually helpful in its current state

  3. ntkathole commented on Sep 18, 2026

    @ntkathole
    Member

    @mao-liu Thank you for the findings, yes, those seems valid bugs. Feel free to tackle and raise issues.

  4. mao-liu commented on Sep 20, 2026

    @mao-liu
    Author

    PR opened! #6853 keen to get your feedback.

    I've also opened #6852 regarding the OpenLineage issue. For my PR, I've deliberately left out any changes to the OpenLineage emitters, since there wouldn't be much value emitting granular PushSource lineage when granular relationships are not emitted for other data sources anyways.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions