Repository navigation
[Bug] [Table model] CASE in a projection over aggregation results corrupts the result block when a later column reuses the same aggregate (Index out of bounds / BufferUnderflowException / Invalid data type) #18792
Description
Search before asking
- I searched in the issues and found nothing similar.
Version
apache/iotdb:2.0.10-standalone(commit3c26382)apache/iotdb:2.0.11-standalone(commit7c0de06)
Both reproduce on a fresh container with default configuration. Table model (-sql_dialect table).
Describe the bug and provide the minimal reproduce step
Description
A GROUP BY query fails when all of the following hold:
- The SELECT list contains a
CASEexpression whoseWHENcondition references an aggregate (e.g.sum(c) > 0); - A column after the CASE references the same aggregate (e.g.
CAST(sum(c) AS DOUBLE)orsum(c) + 0); - The WHERE clause has a predicate that cannot be pushed down to the scan (e.g.
q % 2 < 1), so the plan becomes
scan → filter → partial aggregation → final aggregation → Project, and the Project receives more than one TsBlock; - The groups span more than one batch (here: 2 devices × 2 hourly windows with different shapes).
Depending on the data and the query, the error surfaces as:
301: Index 1 out of bounds for length 1(JDBC / CLI);java.nio.BufferUnderflowException;java.lang.IllegalArgumentException: Invalid data type: DOUBLE/Invalid value: 127(client-side decoding);- in
INSERT INTO … SELECT …the client only gets301: null, while the server log shows
ArrayIndexOutOfBoundsException: Index N out of bounds for length N.
We hit it in a downsampling job (INSERT INTO … SELECT … GROUP BY …, date_bin(…)) computing a weighted average
CASE WHEN sum(cnt) > 0 THEN sum(total) / sum(cnt) END next to sum(cnt), with a quality-mask filter written as
quality % 8 < 4 AND …. Single windows always worked; longer ranges failed deterministically.
Minimal reproduction
3 rows. Full script with controls attached as iotdb-case-repro.sql.
CREATE DATABASE bug_db WITH (TTL='INF');
CREATE TABLE bug_db.t (time TIMESTAMP TIME, dev STRING TAG, c DOUBLE FIELD, q INT32 FIELD);
INSERT INTO bug_db.t (time, dev, c, q) VALUES
(2026-01-01T00:00:00+00:00, 'd0', 1.0, 0),
(2026-01-01T00:00:00+00:00, 'd1', 1.0, 0),
(2026-01-01T01:00:00+00:00, 'd1', 1.0, 0);
SELECT CASE WHEN sum(c) > 0 THEN 1.0 END, CAST(sum(c) AS DOUBLE)
FROM bug_db.t WHERE q % 2 < 1
GROUP BY dev, date_bin(1h, time);
-- Msg: org.apache.iotdb.jdbc.IoTDBSQLException: 301: Index 1 out of bounds for length 1Expected: 3 rows (d0/hour 0, d1/hour 0, d1/hour 1), each 1.0, 1.0.
Controls (each differs from the failing query in exactly one aspect)
| Variant | Result |
|---|---|
| failing query above | 301: Index 1 out of bounds for length 1 |
second column sum(c) + 0 instead of CAST(sum(c) AS DOUBLE) |
301: Index 1 out of bounds for length 1 |
THEN CAST(sum(c) AS DOUBLE) / CAST(sum(c) AS DOUBLE) |
BufferUnderflowException |
| swap the two columns (CASE last) | OK, 3 rows |
second column plain sum(c) (EXPLAIN shows the planner puts the bare symbol before the CASE) |
OK, 3 rows |
| no WHERE | OK, 3 rows |
WHERE q < 1 (pushed down to the scan) |
OK, 3 rows |
GROUP BY dev only |
OK, 2 rows |
EXPLAIN of the failing query shows the Project expressions as
[(CASE WHEN ("sum" > 0) THEN 1E0 END), "sum"] — i.e. CAST(sum(c) AS DOUBLE) is simplified to the same symbol
"sum" that the CASE condition uses, so both outputs share one column transformer.
Server stack trace (2.0.11)
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1
at org.apache.iotdb.calc.transformation.dag.column.leaf.ConstantColumnTransformer.evaluateWithSelection(ConstantColumnTransformer.java:46)
at org.apache.iotdb.calc.transformation.dag.column.AbstractCaseWhenThenColumnTransformer.evaluate(AbstractCaseWhenThenColumnTransformer.java:99)
at org.apache.iotdb.calc.transformation.dag.column.ColumnTransformer.tryEvaluate(ColumnTransformer.java:46)
at org.apache.iotdb.calc.execution.operator.process.FilterAndProjectOperator.getTransformedTsBlock(FilterAndProjectOperator.java:273)
at org.apache.iotdb.calc.execution.operator.process.FilterAndProjectOperator.next(FilterAndProjectOperator.java:133)
at org.apache.iotdb.calc.execution.operator.Operator.nextWithTimer(Operator.java:48)
at org.apache.iotdb.db.queryengine.execution.operator.sink.IdentitySinkOperator.next(IdentitySinkOperator.java:109)
...
The THEN branch is evaluated with a selection array whose length comes from the WHEN column (2 positions),
while the current input block has only 1 position.
What did you expect to see?
Expected: 3 rows (d0/hour 0, d1/hour 0, d1/hour 1), each 1.0, 1.0.
What did you see instead?
error code: 301
Anything else?
No response
Are you willing to submit a PR?
- I'm willing to submit a PR!
Activity
HTHou commentedon Oct 4, 2026
@JackieTien97 PTAL?
JackieTien97 commentedon Oct 6, 2026
I think this is a bug, and I'll fix it soon
JackieTien97 commentedon Oct 7, 2026
The root cause is a cache lifecycle bug in the ColumnTransformer DAG used by FilterAndProjectOperator.
Cached columns are normally released when their reference count reaches zero. However, CASE short-circuit evaluation explicitly clears its child expression caches. If a later projection reuses one of those expressions, recomputation resets its cache reference count to the full number of references in the DAG, without accounting for references already consumed in the current block.
For example, when an aggregate column is referenced by both the CASE condition and a subsequent output:
This mixes columns from different TsBlocks. Different block sizes can cause out-of-bounds access or produce an inconsistent result block, leading to client decoding errors. Equal-sized blocks can also silently return stale values.
This explains why putting CASE last avoids the reported failure: CASE clears the shared cache after the other output has consumed it. Aggregation and the non-pushdown filter expose the problem by producing multiple input blocks; the underlying defect is in expression cache management.
A fix should ensure that expression caches cannot survive across input blocks, including caches of intermediate expressions and descendants of nested CASE expressions, while preserving reuse within the current block.
@ColinLeeo You can take a look and fix that.