Sandbox::https://github.com/feast-dev/feast/pull/5526

ntkathole · 2025-07-20T14:37:54Z

What this PR does / why we need it:

This PR introduces Ray as both a compute engine and an offline store in Feast. It enables scalable, distributed feature engineering and data access in Feast using Ray, making it easier to handle large datasets and complex feature pipelines.

Key Features

Ray Compute Engine

Distributed DAG execution for feature transformations, aggregations, joins, and materialization.
Automatic selection between broadcast and distributed join strategies.
Resource-aware partitioning and parallelism.
Efficient point-in-time joins for historical feature retrieval.
Seamless integration with Feast’s feature view and materialization APIs.

Ray Offline Store

High-performance data reading and writing using Ray Datasets.
Support for local and distributed Ray clusters.
Batch writing and saved dataset persistence.
Configurable resource limits for safe local development and testing.

Documentation added for both Ray compute engine and offline store, including configuration, resource management, and usage examples.

Signed-off-by: ntkathole <nikhilkathole2683@gmail.com>

franciscojavierarceo · 2025-08-11T16:07:20Z

+        timestamp_col = self.column_info.timestamp_column
+        requested_feats = getattr(self.column_info, "feature_cols", [])
+
+        # Check if the feature dataset contains aggregated features (from aggregation node)


franciscojavierarceo

this looks great some small nits but i think we're almost there. can you make some issues to track the follow up here so we don't forget and I'll merge 👍

franciscojavierarceo · 2025-08-11T16:15:37Z

        cls,
-        udf: Callable[[Any], Any],
-        udf_string: str,
+        udf: Optional[Callable[[Any], Any]] = None,


why make this optional?

It's not optional in real use, we are still maintaining strict validation (lines 72-75 enforce that they can't actually be None). We have to make it optional to ensures that PandasTransformation objects can be properly reconstructed across Ray workers without constructor failures.

Signed-off-by: ntkathole <nikhilkathole2683@gmail.com>

franciscojavierarceo · 2025-08-12T17:20:39Z

Thanks for this @ntkathole !!!

# [0.52.0](v0.51.0...v0.52.0) (2025-08-14) ### Bug Fixes * Correct entity value type mapping for aliased feature views ([#5492](#5492)) ([bdf20bb](bdf20bb)) * Correct namespace reference in remote Feast project setup for operator upgrade and previous version tests ([df391ec](df391ec)) * dell pydantic v1 ([1189512](1189512)) * Fixed the entity to on-demand feature view relationship ([1c59bba](1c59bba)) * Make transformers optional ([#5544](#5544)) ([a4eef38](a4eef38)) * Push Source inherits the timestamp fields from Data Source ([#5550](#5550)) ([b7ea5cc](b7ea5cc)) * Remove the devcontainer folder. ([a9815c2](a9815c2)) ### Features * Added API for discovering Feature Views by popular tags ([#5558](#5558)) ([2e5f564](2e5f564)) * Added filtering support for featureView and featureServices api ([#5552](#5552)) ([897b3f3](897b3f3)) * Added global search api and necessary unit tests ([#5532](#5532)) ([dd3061f](dd3061f)) * Added Ray Compute Engine and Ray Offline Store Support ([#5526](#5526)) ([72de088](72de088)) * Added recent visit logging api for registry server ([#5545](#5545)) ([2adcf2c](2adcf2c)) * **auth:** support client-credentials & static token for OIDC client auth ([fc44222](fc44222)) * **auth:** support client-credentials & static token for OIDC client auth ([795fc06](795fc06)) * Implement and enhance remote document retrieval functionality ([#5487](#5487)) ([d095b96](d095b96)) * Implemented consistent error handling ([7f10151](7f10151)) * Offline Store historical features retrieval without entity df, but based on datatime range ([#5527](#5527)) ([df942b9](df942b9))

ntkathole self-assigned this Jul 20, 2025

ntkathole requested a review from a team as a code owner July 20, 2025 14:37

ntkathole force-pushed the ray_compute branch from 3a142ba to 38850cc Compare July 20, 2025 16:47

ntkathole added the ok-to-test label Jul 20, 2025

ntkathole marked this pull request as draft July 21, 2025 05:18

haoxu0 reviewed Jul 22, 2025

View reviewed changes

Comment thread sdk/python/feast/infra/compute_engines/ray/feature_builder.py

ntkathole force-pushed the ray_compute branch 20 times, most recently from 8549556 to 76af065 Compare July 26, 2025 13:14

ntkathole marked this pull request as ready for review July 26, 2025 13:16

ntkathole requested review from franciscojavierarceo and haoxu0 July 26, 2025 14:28

haoxu0 reviewed Jul 26, 2025

View reviewed changes

Comment thread sdk/python/feast/infra/compute_engines/ray/feature_builder.py Outdated

ntkathole force-pushed the ray_compute branch from 05d9827 to cf0e01f Compare August 11, 2025 04:52

franciscojavierarceo reviewed Aug 11, 2025

View reviewed changes

Comment thread sdk/python/feast/infra/compute_engines/ray/compute.py Outdated

ntkathole added 9 commits August 11, 2025 21:09

feat: Added ray to requirements

f1e78e9

Signed-off-by: ntkathole <nikhilkathole2683@gmail.com>

feat: Added Ray offline store

bd27d23

Signed-off-by: ntkathole <nikhilkathole2683@gmail.com>

feat: Improved Ray distributed processing

51c3426

Signed-off-by: ntkathole <nikhilkathole2683@gmail.com>

Fixed tests

5b629b4

Signed-off-by: ntkathole <nikhilkathole2683@gmail.com>

fix: Added utils for validation

a806ae3

Signed-off-by: ntkathole <nikhilkathole2683@gmail.com>

feat: Use Ray Dataset for data processing

b31a8fd

Signed-off-by: ntkathole <nikhilkathole2683@gmail.com>

feat: Added Ray Compute Engine

266f4cd

Signed-off-by: ntkathole <nikhilkathole2683@gmail.com>

fix: Fixed logic for source/derived feature views

1da9c94

Signed-off-by: ntkathole <nikhilkathole2683@gmail.com>

fix: Remove redundant source resolution

c3a87ab

Signed-off-by: ntkathole <nikhilkathole2683@gmail.com>

franciscojavierarceo reviewed Aug 11, 2025

View reviewed changes

Comment thread sdk/python/feast/infra/compute_engines/ray/feature_builder.py

franciscojavierarceo reviewed Aug 11, 2025

View reviewed changes

Comment thread sdk/python/feast/infra/compute_engines/ray/feature_builder.py

franciscojavierarceo reviewed Aug 11, 2025

View reviewed changes

Comment thread sdk/python/feast/infra/compute_engines/ray/job.py

franciscojavierarceo reviewed Aug 11, 2025

View reviewed changes

Comment thread sdk/python/feast/infra/compute_engines/ray/nodes.py Outdated

franciscojavierarceo reviewed Aug 11, 2025

View reviewed changes

Comment thread sdk/python/feast/infra/offline_stores/contrib/ray_offline_store/ray.py Outdated

franciscojavierarceo reviewed Aug 11, 2025

View reviewed changes

Comment thread sdk/python/feast/infra/offline_stores/contrib/ray_offline_store/ray.py

franciscojavierarceo reviewed Aug 11, 2025

View reviewed changes

ntkathole force-pushed the ray_compute branch from cf0e01f to af613d4 Compare August 12, 2025 04:08

ntkathole mentioned this pull request Aug 12, 2025

Generalize build materialization_plan from the FeatureBuilder to store intermediate results #5566

Open

fix: Handle comments and removed use_ray_cluster config

5435859

Signed-off-by: ntkathole <nikhilkathole2683@gmail.com>

ntkathole force-pushed the ray_compute branch from af613d4 to 5435859 Compare August 12, 2025 05:08

ntkathole requested a review from franciscojavierarceo August 12, 2025 05:35

franciscojavierarceo approved these changes Aug 12, 2025

View reviewed changes

franciscojavierarceo merged commit 72de088 into feast-dev:master Aug 12, 2025

ntkathole mentioned this pull request Aug 16, 2025

to_ray_dataset as a first citizen method for retrieval_job #5568

Closed

franciscojavierarceo mentioned this pull request Sep 11, 2026

docs: Add Nikhil Kathole as a project maintainer #6830

Merged

3 tasks done

Navigation Menu

Repository navigation

feat: Added Ray Compute Engine and Ray Offline Store Support - #5526

feat: Added Ray Compute Engine and Ray Offline Store Support#5526
franciscojavierarceo merged 10 commits into
feast-dev:masterfrom
ntkathole:ray_compute

ntkathole commented Jul 20, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

franciscojavierarceo Aug 11, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

franciscojavierarceo left a comment

Uh oh!

Uh oh!

franciscojavierarceo Aug 11, 2025

Uh oh!

ntkathole Aug 12, 2025

Uh oh!

franciscojavierarceo commented Aug 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

ntkathole commented Jul 20, 2025

What this PR does / why we need it:

Key Features

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

franciscojavierarceo Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

franciscojavierarceo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

franciscojavierarceo Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

ntkathole Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

franciscojavierarceo commented Aug 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants