Visitar URL original
feat: Support Feast DQM for Chronon outputs with freshness and lineage context · Issue #6984 · feast-dev/feast · GitHub
Skip to content

feat: Support Feast DQM for Chronon outputs with freshness and lineage context #6984

Description

@franciscojavierarceo

Problem

Teams should be able to inspect Chronon feature quality in their existing Feast project. Our working demonstration currently profiles an exported checkout snapshot using a separate Dask/FileSource setup. That export's timestamp describes the snapshot, not freshness of Chronon's source data or serving pipeline.

The Chronon adapter can already read published Parquet for profiling, and Feast has a Python metric-computation fallback. The missing integration includes persistent monitoring storage: the adapter does not implement the monitoring storage hooks expected by MonitoringService. Online reads also currently return no feature timestamp, so they cannot establish source freshness.

Proposed solution

Start with persisted DQM over Chronon batch outputs in a Chronon-configured Feast repository:

  • Implement the required metrics, baseline, and monitoring-job persistence, either through the adapter or an explicitly configured metrics backend.
  • Reuse the existing Feast monitoring API/UI for profiles and historical date buckets.
  • Associate monitored outputs with the Feast FeatureView and Chronon GroupBy/Join, plus output version or pipeline run when available.
  • Keep event time, feature-computation time, export time, and metric-computation time distinct. Missing timestamp or provenance metadata should remain unknown.

Continuous serving-log ingestion and online/offline consistency checks should have separate follow-ups unless explicitly implemented here. Output profiling alone does not establish those properties.

Acceptance criteria

  • One Chronon-configured repository computes and displays profiles, baselines, and multiple date buckets through the existing REST API/UI, without duplicating definitions in a separate Dask/FileSource repository.
  • Metrics and job state survive restart, retries do not create duplicate metrics, and baseline replacement works.
  • Integration fixtures verify null preservation, mapped feature names, entity keys, timestamp filtering, and known metric values.
  • A stale-output fixture with a recent export is not reported as source-fresh. Absent timestamp/run metadata has an explicit unknown state.
  • API/UI metadata lets users trace a monitored feature to its Chronon definition and available output provenance.
  • Documentation and a runnable example explain supported storage, permissions, date semantics, and the distinction between output quality, pipeline freshness, and online/offline consistency.

Alternatives considered

Continue exporting snapshots into a separate monitoring repository, or use an external quality tool. A configurable metrics backend may be preferable to forcing monitoring writes into Chronon's feature storage; evaluate that choice in the design.

Context

Follow-up to #6188; related definition and execution metadata work: #6982 and #6983.

Starting points: Chronon offline adapter, monitoring storage contract, MonitoringService, and Chronon online adapter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions