Repository navigation
Spark Materialization Engine Cannot Infer Schema #5594
Description
Activity
Note: I've tested this with
pyspark==4.0.0and it works, I believe this is due to changes in the way Spark now natively uses arrow. Unfortunately we're still blocked though as Spark 4 isn't widely available on most data compute platforms (e.g. AWS EMR) yet.I believe converting the pyarrow table to a pandas DataFrame as an intermediary should resolve this which shouldn't be too memory intensive if
spark.sql.execution.arrow.enabledis set totrueReacted by Jacob WeinholdThat make sense. Feel free to submit a pr
The reproduction repository passes now, using your suggested fix @danbaron63
- added a commit that references this issue
on Jan 16, 2026 - added a commit that references this issue
on Jan 21, 2026 - added 2 commits that reference this issue
on Feb 7, 2026
Expected Behavior
Spark engine when enabled should materialize features to the online store.
Current Behavior
Materialization fails with the following error:
Steps to reproduce
See this repo that should reproduce the error locally for you. Ensure your
JAVA_HOMEis pointing to a java 17 installation and runmake runto run the workflow. Optionally inspect the Makefile and run the commands yourself.The repo has the default
feast initrepo created and will run afeast plan,feast applyand finally amaterialize.pyscript which should trigger a Spark materialization job.Specifications
Possible Solution
I believe this happens when an arrow dataframe is converted to a Spark dataframe. Why this happens I am unsure, it may be my configuration or it may be a bug. If someone could advise on a solution that would be great.
Thanks!