Repository navigation
[Spark 4.2 UT] Umbrella: re-enable disabled Spark 4.2 unit tests (GLUTEN-12569) #13181
Description
Umbrella tracking issue for re-enabling the Spark 4.2 unit tests that are currently disabled to keep the gluten-ut/spark42 CI green.
Added in PR #13163 (GLUTEN-12569). 150 UTs are disabled across 7 root-cause themes. Each theme has its own sub-issue with the exact disabled tests, the disable mechanism, and the files updated. As the underlying Gluten/Velox support lands, remove the corresponding .exclude(...) / allowlist / @Ignore / ignore(...) entries and close the sub-issue.
Sub-issues (7)
- [Spark 4.2 UT] Spark 4.2 Storage-Partitioned Join (SPJ) support in Gluten #13174 — Spark 4.2 Storage-Partitioned Join (SPJ) support in Gluten (29 tests)
- [Spark 4.2 UT] Spark 4.2 DSv2 row-level writes (DELETE/MERGE/UPDATE/INSERT) on Gluten path #13175 — Spark 4.2 DSv2 row-level writes (DELETE/MERGE/UPDATE/INSERT) on Gluten path (39 tests)
- [Spark 4.2 UT] Floating-point collection semantics: -0.0/NaN normalization & RESPECT NULLS #13176 — Floating-point collection semantics: -0.0/NaN normalization & RESPECT NULLS (12 tests)
- [Spark 4.2 UT] Golden-file drift on Spark 4.2 (TPC-DS plan-stability + SQLQueryTest .sql) #13177 — Golden-file drift on Spark 4.2 (TPC-DS plan-stability + SQLQueryTest .sql) (46 tests)
- [Spark 4.2 UT] Velox expression & operator gaps (math/TIME/date/broadcast/hash-agg) #13178 — Velox expression & operator gaps (math/TIME/date/broadcast/hash-agg) (6 tests)
- [Spark 4.2 UT] Spark 4.2 framework/API changes breaking Gluten tests (non-velox) #13179 — Spark 4.2 framework/API changes breaking Gluten tests (non-velox) (7 tests)
- [Spark 4.2 UT] File-source & Hive scan (Parquet/ORC ignoreMissingFiles/NullType, bucketed-scan AE, ADD JAR) #13180 — File-source & Hive scan (Parquet/ORC ignoreMissingFiles/NullType, bucketed-scan AE, ADD JAR) (11 tests)
Summary
| # | Theme | Items | Sub-issue |
|---|---|---|---|
| 1 | Spark 4.2 Storage-Partitioned Join (SPJ) support in Gluten | 29 | #13174 |
| 2 | Spark 4.2 DSv2 row-level writes (DELETE/MERGE/UPDATE/INSERT) on Gluten path | 39 | #13175 |
| 3 | Floating-point collection semantics: -0.0/NaN normalization & RESPECT NULLS | 12 | #13176 |
| 4 | Golden-file drift on Spark 4.2 (TPC-DS plan-stability + SQLQueryTest .sql) | 46 | #13177 |
| 5 | Velox expression & operator gaps (math/TIME/date/broadcast/hash-agg) | 6 | #13178 |
| 6 | Spark 4.2 framework/API changes breaking Gluten tests (non-velox) | 7 | #13179 |
| 7 | File-source & Hive scan (Parquet/ORC ignoreMissingFiles/NullType, bucketed-scan AE, ADD JAR) | 11 | #13180 |
| Total | 150 |
How these were disabled
.exclude(...)ingluten-ut/spark42/.../velox/VeloxTestSettings.scala(most suites, honoringBackendTestSettings.shouldRun)- allowlist removal in
gluten-ut/spark42/.../velox/VeloxSQLQueryTestSettings.scala(GlutenSQLQueryTestSuite.sqlgolden tests — that suite ignores the exclude registry) - class-level
@org.scalatest.Ignorefor whole-suite aborts (GlutenBloomFilterFallbackSuite,MiscOperatorSuite,GlutenWholeStageCodegenSuite) test(...)->ignore(...)for one standalone test (GlutenAutoAdjustStageResourceProfileSuite)
All disables are CI-confirmed (local-only false-positives were removed via a validation run with excludes reverted).
Related: PR #13163, GLUTEN-12569.
Activity