Repository navigation
Release DataFusion 54.2.0 (minor/patch) Release #23805
Description
Is your feature request related to a problem or challenge?
- Part of [EPIC] DataFusion Releases #19783
- Follow on to Release DataFusion
54.1.0(minor/patch) Release #22547
Let's plan a 54.1.1 or 54.2.0 patch release to fix any bugs / issues found in 54.1.0.
Steps:
- Update version and changelog
- Test with DataFusion Python
- Test with DataFusion Comet
- Test with delta.rs:
- Test vortex:
- Test with iceberg-rust:
- Test with LakeSail:
- Test with parquet viewer
- Test with datafusion-materialized-views
- Voting Thread:
- Publish to crates.io
Potential Content:
Bug Fixes
- fix: NOT IN with NULL subquery returns wrong results under SortMergeJoin #22810
- DataFusion 54.1.0 has a breaking API change #23886
-
RepartitionExecwithholds all output from an unbounded input untilbatch_sizerows accumulate #24044 - fix: prevent LEAD/LAG IGNORE NULLS panic without null bitmap #23706 from @xudong963 (requested by @lilic); Backport PR: [branch-54] fix: prevent LEAD/LAG IGNORE NULLS panic without null bitmap (#23706) #26319
- fix: keep a CoalescePartitionsExec required by a SinglePartition child #23948 from @adriangb (requested by @hareshkh); Backport PR: [branch-54] fix: keep a CoalescePartitionsExec required by a SinglePartition child #26961
- fix: read Hive's null partition placeholder as NULL #26310 from @xanderbailey (requested by @hareshkh); Backport PR: [branch-54] fix: read Hive's null partition placeholder as NULL (backport #26310) #27308
- Expand scalar async UDF results to the batch row count #25828 from @hareshkh (requested by @kamilk-pal); Backport PR: [branch-54] Expand scalar async UDF results to the batch row count (#25828) #27372
- fix(smj): handle floating-point keys across batch boundaries #25134 from @jeffw13; Backport PR: [branch-54] fix(smj): handle floating-point keys across batch boundaries (#25134) #27357 (@Sunninsky)
- fix: allow NULL-padded rows in sort-merge join filter evaluation #26027 from @CuteChuanChuan; Backport PR: [branch-54] fix: allow NULL-padded rows in sort-merge join filter evaluation (#26027) #27356 (@Sunninsky)
@alamb Declined to backport as it wasn't a clean backport
- fix: ensure deferred-filtered outer joins preserve streamed output order #24573 from @jayzhan211; Backport PR: [branch-54] fix: ensure deferred-filtered outer joins preserve streamed output order (#24573) #27355 (@Sunninsky)
Performance
- TBD
Other
- TBD
Minor Features:
- TBD
Activity
AdamGS commentedon Jul 22, 2026
Would love to get this fix in, happy to backport it once its merged to
main.andygrove commentedon Jul 25, 2026
DataFusion 54.1.0 had a breaking API change compared to 54.0.0, which caused issues for some downstream projects. For example, here is the PR to fix data fusion-python apache/datafusion-python#1645
I wonder if we should consider reverting the breaking change for the 54.2.0 release?
lilic commentedon Oct 14, 2026
@mbutrovich 👋 hello, I am just checking if this patch release is still planned to be cut? Thanks!
mbutrovich commentedon Oct 14, 2026
I'm not sure we had a volunteer to run the release, but I can probably do it. I will raise it at the DataFusion Community Meeting today.
lilic commentedon Oct 14, 2026
@mbutrovich thank you much appreciated, let me know if there is any way I can help!
mbutrovich commentedon Oct 14, 2026
Is there a particular fix you're on the lookout for to make sure it gets backported?
lilic commentedon Oct 14, 2026
Thanks! The one that would be nice to have for us is the LEAD/LAG IGNORE NULLS panic fix #23706. It's in 55 but not 54. From what I can tell should be easy to port, but unsure of the rules of how you all decide on what can go in. I am happy to open the PR if that makes it easier.
alamb commentedon Oct 14, 2026
Apparently Comet has already moved on to 55, so they are no longer waiting on a patch release of 54
However I suspect there may be others who are interested in a patch release too and I am happy to run the release if so.
@lilic Please do make a backport for #23706 and ping me and I'll be happy to review/merge it
I posted a note to the dev list too asking for other requests
lilic commentedon Oct 14, 2026
@alamb thanks, I opened #26319.
[branch-54] fix: prevent LEAD/LAG IGNORE NULLS panic without null bit…
hareshkh commentedon Oct 28, 2026
@alamb @adriangb:
I am happy to cherry-pick to branch-54
alamb commentedon Oct 28, 2026
Pease just make the Prs to the branch and tag me -- I can help merge
[branch-54] fix: keep a CoalescePartitionsExec required by a SinglePa…
hareshkh commentedon Nov 4, 2026
hareshkh commentedon Nov 9, 2026
alamb commentedon Nov 9, 2026
Sorry @hareshkh -- I have been trying to help the 56 release.
If you think we should backport #26310 please just open a backport PR
Otherwise, is there any other content we need before we make this release candidate?
kamilk-pal commentedon Nov 12, 2026
Could we get included #27372 in the release too?
[branch-54] fix: read Hive's null partition placeholder as NULL (back…
[branch-54] Expand scalar async UDF results to the batch row count (#…
[branch-54] fix(smj): handle floating-point keys across batch boundar…
[branch-54] fix: allow NULL-padded rows in sort-merge join filter eva…
alamb commentedon Nov 12, 2026
PR to update version
[branch-54] chore: fix cargo audit (backport #24467, #25309, #26253, #…