Reusing Entities across FeatureViews with different source data column names #4840

martinpella · 2024-12-11T17:37:35Z

martinpella
Dec 11, 2024

Hello,
I want to define an Entity that can be reused across multiple FeatureViews. Here is an example:

customer_entity = Entity(
    name="customer_id",
    join_keys=["customer_id"],
    value_type=ValueType.INT64,
)

The challenge arises because not all source tables have a uniform column name for customer_id; for instance, some tables might use user_id, customer, etc.

Consider a scenario where we want to utilize the above-defined entity for a FeatureView with a BigQuery table that uses user_id as a column name instead of customer_id:

fv = FeatureView(
    name="test__features",
    entities=[customer_entity],
    ttl=timedelta(0),
    schema=[
        Field(name="customer_id", dtype=Int64),
        Field(name="feature1", dtype=Float64),
        Field(name="feature2", dtype=Float64),
    ],
    online=False,
    source=BigQuerySource(
        table="catalog.schema.table1",
        timestamp_field="timestamp_column",
        field_mapping={
            "feature1_column_name": "feature1",
            "feature2_column_name": "feature2",
            "user_id": "customer_id",
        },
    ),
)

The specific issue occurs in the subpart of the generated query here, where the query tries to select customer_id rather than user_id leading to the following error: google.api_core.exceptions.BadRequest: 400 Query error: Unrecognized name: customer_id

test__features__subquery AS (
        SELECT
                timestamp_column as event_timestamp,
            
                customer_id AS customer_id,
            
                feature1_column_name as feature1, 
            
                feature2_column_name as feature2
            
        FROM `catalog.schema.table1`
        WHERE P_CREATION_DATE <= '2024-12-09T00:00:00'
    ),

I've also explored the option of Entity aliasing:

fs = FeatureService(
    name="test_service",
    features=[fv.with_name("test").with_join_key_map({"customer_id": "user_id"})],
)

Although this approach results in a different subquery, the issue persists where the query incorrectly attempts to select customer_id from the table:

test__subquery AS (
        SELECT
                timestamp_column as event_timestamp,
            
                customer_id AS user_id,
            
                feature1_column_name as feature1, 
            
                feature2_column_name as feature2
            
        FROM `catalog.schema.table1`
        WHERE P_CREATION_DATE <= '2024-12-09T00:00:00'

I experimented with reversing the key/value pair in the join map, but unfortunately, the problem still remains.

Any suggestions or insights on how to address this mismatch would be greatly appreciated.

Navigation Menu

Repository navigation

Reusing Entities across FeatureViews with different source data column names #4840

Uh oh!

{{title}}

Uh oh!

Replies: 0 comments

Select a reply

Uh oh!

Reusing Entities across FeatureViews with different source data column names #4840

Uh oh!

martinpella Dec 11, 2024

Replies: 0 comments

martinpella
Dec 11, 2024