Visitar URL original
[branch-54] fix: prevent LEAD/LAG IGNORE NULLS panic without null bit… · apache/datafusion@1e317c7 · GitHub
Skip to content

Commit 1e317c7

Browse files
lilicxudong963
andauthored
[branch-54] fix: prevent LEAD/LAG IGNORE NULLS panic without null bitmap (#23706) (#26319)
- part of #23805 Backport of #23706 by @xudong963. ## Which issue does this PR close? Part of #23805, originally closed #23705. ## Rationale for this change Arrow arrays containing no null values commonly omit the null bitmap. The whole-partition evaluation path for `LEAD` and `LAG` with `IGNORE NULLS` unconditionally unwrapped that optional bitmap, causing a panic for valid non-null input arrays. ## What changes are included in this PR? - Fall back to the existing regular shift implementation when the input has no null bitmap. - Add a regression test covering both `LEAD` and `LAG` with an array that explicitly has no null bitmap. ## Are these changes tested? Yes. ## Are there any user-facing changes? `LEAD` and `LAG` with `IGNORE NULLS` no longer panic when the input Arrow array has no null bitmap. There are no public API changes. (cherry picked from commit a8d1af6) Co-authored-by: xudong.w <wxd963996380@gmail.com>
1 parent 0d1f2eb commit 1e317c7

1 file changed

Lines changed: 27 additions & 2 deletions

File tree

‎datafusion/functions-window/src/lead_lag.rs‎

Lines changed: 27 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -433,8 +433,12 @@ fn evaluate_all_with_ignore_null(
433433
default_value: &ScalarValue,
434434
is_lag: bool,
435435
) -> Result<ArrayRef, DataFusionError> {
436-
let valid_indices: Vec<usize> =
437-
array.nulls().unwrap().valid_indices().collect::<Vec<_>>();
436+
// Arrays without NULLs do not necessarily have a null bitmap.
437+
let Some(nulls) = array.nulls() else {
438+
return shift_with_default_value(array, offset, default_value);
439+
};
440+
441+
let valid_indices: Vec<usize> = nulls.valid_indices().collect::<Vec<_>>();
438442
let direction = !is_lag;
439443
let new_array_results: Result<Vec<_>, DataFusionError> = (0..array.len())
440444
.map(|id| {
@@ -838,4 +842,25 @@ mod tests {
838842
.collect::<Int32Array>(),
839843
)
840844
}
845+
846+
#[test]
847+
fn test_ignore_nulls_without_null_bitmap() -> Result<()> {
848+
let input = Int32Array::from(vec![1, 2, 3]);
849+
assert!(input.nulls().is_none());
850+
let input: ArrayRef = Arc::new(input);
851+
852+
for (offset, expected) in [
853+
(1, Int32Array::from(vec![None, Some(1), Some(2)])),
854+
(-1, Int32Array::from(vec![Some(2), Some(3), None])),
855+
] {
856+
let actual = evaluate_all_with_ignore_null(
857+
&input,
858+
offset,
859+
&ScalarValue::Int32(None),
860+
offset > 0,
861+
)?;
862+
assert_eq!(expected, *as_int32_array(&actual)?);
863+
}
864+
Ok(())
865+
}
841866
}

0 commit comments

Comments
 (0)