refactor: Replace combinatorial nested collection enums with recursiv…

…e VALUE_LIST/VALUE_SET Replace 4 combinatorial enum values (LIST_LIST=36, LIST_SET=37, SET_LIST=38, SET_SET=39) with 2 recursive enum values (VALUE_LIST=40, VALUE_SET=41) that use RepeatedValue to enable unlimited nesting depth. This is a breaking change for an unreleased feature, as suggested in PR #6132 review. Key changes: - Proto: Remove 4 enum/oneof fields, add VALUE_LIST/VALUE_SET with reserved 36-39 - Python: Update ValueType enum, type system, serialization, field persistence - JSON: Update proto_json encode/decode for new field names - Tests: Rewrite all nested collection tests (204 tests passing) - Docs: Update type-system.md for recursive design Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: soojin <soojin@dable.io>
franciscojavierarceo · Apr 2, 2026 · Mar 17, 2026 · Mar 18, 2026 · Mar 18, 2026 · Mar 19, 2026
commit fd0d745fcab9c78957aacabf8dcb4baf442f041b
@@ -88,21 +88,21 @@ All primitive types (except `Map` and `Json`) have corresponding set types for s
 
 ### Nested Collection Types
 
-Feast supports 2-level nested collections, combining Array and Set types:
+Feast supports arbitrarily nested collections using a recursive `VALUE_LIST` / `VALUE_SET` design. The outer container determines the proto enum (`VALUE_LIST` for `Array(…)`, `VALUE_SET` for `Set(…)`), while the full inner type structure is persisted via a mandatory `feast:nested_inner_type` Field tag.
 
 | Feast Type | Python Type | ValueType | Description |
 |------------|-------------|-----------|-------------|
-| `Array(Array(T))` | `List[List[T]]` | `LIST_LIST` | List of lists |
-| `Array(Set(T))` | `List[List[T]]` | `LIST_SET` | List of sets (inner elements deduplicated) |
-| `Set(Array(T))` | `List[List[T]]` | `SET_LIST` | Set of lists |
-| `Set(Set(T))` | `List[List[T]]` | `SET_SET` | Set of sets (inner elements deduplicated) |
+| `Array(Array(T))` | `List[List[T]]` | `VALUE_LIST` | List of lists |
+| `Array(Set(T))` | `List[List[T]]` | `VALUE_LIST` | List of sets |
+| `Set(Array(T))` | `List[List[T]]` | `VALUE_SET` | Set of lists |
+| `Set(Set(T))` | `List[List[T]]` | `VALUE_SET` | Set of sets |
+| `Array(Array(Array(T)))` | `List[List[List[T]]]` | `VALUE_LIST` | 3-level nesting |
 
-Where `T` is any supported primitive type (Int32, Int64, Float32, Float64, String, Bytes, Bool, UnixTimestamp).
+Where `T` is any supported primitive type (Int32, Int64, Float32, Float64, String, Bytes, Bool, UnixTimestamp) or another nested collection type.
 
 **Notes:**
-- Nesting is limited to 2 levels. `Array(Array(Array(T)))` will raise a `ValueError`.
-- Inner type information is preserved via Field tags (`feast:nested_inner_type`) and restored during deserialization.
-- For `Array(Set(T))` and `Set(Set(T))`, inner collection elements are automatically deduplicated.
+- Nesting depth is **unlimited**. `Array(Array(Array(T)))`, `Set(Array(Set(T)))`, etc. are all supported.
+- Inner type information is preserved via Field tags (`feast:nested_inner_type`) and restored during deserialization. This tag is mandatory for nested collection types.
 - Empty inner collections (`[]`) are stored as empty proto values and round-trip as `None`. For example, `[[1, 2], [], [3]]` becomes `[[1, 2], None, [3]]` after a write-read cycle.
 
 ### Map Types
@@ -315,23 +315,20 @@ unique_devices = {uuid.uuid4(), uuid.uuid4()}
 
 ### Nested Collection Type Usage Examples
 
-Nested collections allow storing multi-dimensional data:
+Nested collections allow storing multi-dimensional data with unlimited depth:
 
 ```python
 # List of lists — e.g., weekly score history per user
 weekly_scores = [[85.0, 90.5, 78.0], [92.0, 88.5], [95.0, 91.0, 87.5]]
 
 # List of sets — e.g., unique tags assigned per category
 unique_tags_per_category = [["python", "ml"], ["rust", "systems"], ["python", "web"]]
-# Inner sets are automatically deduplicated:
-# [["python", "ml"], ...] (duplicates within each inner set are removed)
 
-# Set of lists — e.g., distinct ordered sequences observed
-distinct_sequences = [[1, 2, 3], [4, 5], [1, 2, 3]]
+# 3-level nesting — e.g., multi-dimensional matrices
+Field(name="tensor", dtype=Array(Array(Array(Float64))))
 
-# Set of sets — e.g., distinct groups of unique items
-distinct_groups = [["a", "b"], ["c", "d"], ["a", "b"]]
-# Inner elements are deduplicated within each set
+# Mixed nesting
+Field(name="grouped_tags", dtype=Array(Set(Array(String))))
 ```
 
 **Limitation:** Empty inner collections round-trip as `None`:

@@ -63,10 +63,8 @@ message ValueType {
     TIME_UUID_LIST = 39;
     UUID_SET = 40;
     TIME_UUID_SET = 41;
-    LIST_LIST = 42;
-    LIST_SET = 43;
-    SET_LIST = 44;
-    SET_SET = 45;
+    VALUE_LIST = 42;
+    VALUE_SET = 43;
   }
 }
 
@@ -112,10 +110,8 @@ message Value {
     StringList time_uuid_list_val = 39;
     StringSet uuid_set_val = 40;
     StringSet time_uuid_set_val = 41;
-    RepeatedValue list_list_val = 42;
-    RepeatedValue list_set_val = 43;
-    RepeatedValue set_list_val = 44;
-    RepeatedValue set_set_val = 45;
+    RepeatedValue list_val = 42;
+    RepeatedValue set_val = 43;
   }
 }
 

@@ -171,13 +171,7 @@ def from_proto(cls, field_proto: FieldProto):
             dtype = Array(inner_struct)
             user_tags = {k: v for k, v in tags.items() if k not in internal_tags}
         elif (
-            value_type
-            in (
-                ValueType.LIST_LIST,
-                ValueType.LIST_SET,
-                ValueType.SET_LIST,
-                ValueType.SET_SET,
-            )
+            value_type in (ValueType.VALUE_LIST, ValueType.VALUE_SET)
             and NESTED_COLLECTION_INNER_TYPE_TAG in tags
         ):
             dtype = _str_to_feast_type(tags[NESTED_COLLECTION_INNER_TYPE_TAG])

@@ -108,7 +108,7 @@ def _proto_value_to_transport_value(proto_value: ValueProto) -> Any:
 
         # Nested collection types use feast_value_type_to_python_type
         # which handles recursive conversion of RepeatedValue protos.
-        if val_attr in ("list_list_val", "list_set_val", "set_list_val", "set_set_val"):
+        if val_attr in ("list_val", "set_val"):
             return feast_value_type_to_python_type(proto_value)
 
         # Map/Struct types are converted to Python dicts by

@@ -63,7 +63,7 @@ def to_json_object(printer: _Printer, message: ProtoMessage) -> JsonObject:
         # to JSON. The parse back result will be different from original message.
         if which is None or which == "null_val":
             return None
-        elif which in ("list_list_val", "list_set_val", "set_list_val", "set_set_val"):
+        elif which in ("list_val", "set_val"):
             # Nested collection: RepeatedValue containing Values
             repeated = getattr(message, which)
             value = [
@@ -96,13 +96,13 @@ def from_json_object(
                 # Nested collection (list of lists).
                 # Check any() to handle cases where the first element is None
                 # (empty inner collections round-trip through proto as None).
-                # Default to list_list_val since JSON transport loses the
+                # Default to list_val since JSON transport loses the
                 # outer/inner set distinction.
                 rv = RepeatedValue()
                 for inner in value:
                     inner_val = rv.val.add()
                     from_json_object(parser, inner, inner_val)
-                message.list_list_val.CopyFrom(rv)
+                message.list_val.CopyFrom(rv)
             elif isinstance(value[0], bool):
                 message.bool_list_val.val.extend(value)
             elif isinstance(value[0], str):