Visitar URL original
[Bug] [Table model] CASE in a projection over aggregation results corrupts the result block when a later column reuses the same aggregate (Index out of bounds / BufferUnderflowException / Invalid data type) · Issue #18792 · apache/iotdb · GitHub
Skip to content

[Bug] [Table model] CASE in a projection over aggregation results corrupts the result block when a later column reuses the same aggregate (Index out of bounds / BufferUnderflowException / Invalid data type) #18792

Description

@tony-tang

Search before asking

  • I searched in the issues and found nothing similar.

Version

  • apache/iotdb:2.0.10-standalone (commit 3c26382)
  • apache/iotdb:2.0.11-standalone (commit 7c0de06)

Both reproduce on a fresh container with default configuration. Table model (-sql_dialect table).

Describe the bug and provide the minimal reproduce step

Description

A GROUP BY query fails when all of the following hold:

  1. The SELECT list contains a CASE expression whose WHEN condition references an aggregate (e.g. sum(c) > 0);
  2. A column after the CASE references the same aggregate (e.g. CAST(sum(c) AS DOUBLE) or sum(c) + 0);
  3. The WHERE clause has a predicate that cannot be pushed down to the scan (e.g. q % 2 < 1), so the plan becomes
    scan → filter → partial aggregation → final aggregation → Project, and the Project receives more than one TsBlock;
  4. The groups span more than one batch (here: 2 devices × 2 hourly windows with different shapes).

Depending on the data and the query, the error surfaces as:

  • 301: Index 1 out of bounds for length 1 (JDBC / CLI);
  • java.nio.BufferUnderflowException;
  • java.lang.IllegalArgumentException: Invalid data type: DOUBLE / Invalid value: 127 (client-side decoding);
  • in INSERT INTO … SELECT … the client only gets 301: null, while the server log shows
    ArrayIndexOutOfBoundsException: Index N out of bounds for length N.

We hit it in a downsampling job (INSERT INTO … SELECT … GROUP BY …, date_bin(…)) computing a weighted average
CASE WHEN sum(cnt) > 0 THEN sum(total) / sum(cnt) END next to sum(cnt), with a quality-mask filter written as
quality % 8 < 4 AND …. Single windows always worked; longer ranges failed deterministically.

Minimal reproduction

3 rows. Full script with controls attached as iotdb-case-repro.sql.

CREATE DATABASE bug_db WITH (TTL='INF');
CREATE TABLE bug_db.t (time TIMESTAMP TIME, dev STRING TAG, c DOUBLE FIELD, q INT32 FIELD);
INSERT INTO bug_db.t (time, dev, c, q) VALUES
  (2026-01-01T00:00:00+00:00, 'd0', 1.0, 0),
  (2026-01-01T00:00:00+00:00, 'd1', 1.0, 0),
  (2026-01-01T01:00:00+00:00, 'd1', 1.0, 0);

SELECT CASE WHEN sum(c) > 0 THEN 1.0 END, CAST(sum(c) AS DOUBLE)
FROM bug_db.t WHERE q % 2 < 1
GROUP BY dev, date_bin(1h, time);
-- Msg: org.apache.iotdb.jdbc.IoTDBSQLException: 301: Index 1 out of bounds for length 1

Expected: 3 rows (d0/hour 0, d1/hour 0, d1/hour 1), each 1.0, 1.0.

Controls (each differs from the failing query in exactly one aspect)

Variant Result
failing query above 301: Index 1 out of bounds for length 1
second column sum(c) + 0 instead of CAST(sum(c) AS DOUBLE) 301: Index 1 out of bounds for length 1
THEN CAST(sum(c) AS DOUBLE) / CAST(sum(c) AS DOUBLE) BufferUnderflowException
swap the two columns (CASE last) OK, 3 rows
second column plain sum(c) (EXPLAIN shows the planner puts the bare symbol before the CASE) OK, 3 rows
no WHERE OK, 3 rows
WHERE q < 1 (pushed down to the scan) OK, 3 rows
GROUP BY dev only OK, 2 rows

EXPLAIN of the failing query shows the Project expressions as
[(CASE WHEN ("sum" > 0) THEN 1E0 END), "sum"] — i.e. CAST(sum(c) AS DOUBLE) is simplified to the same symbol
"sum" that the CASE condition uses, so both outputs share one column transformer.

Server stack trace (2.0.11)

Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1
  at org.apache.iotdb.calc.transformation.dag.column.leaf.ConstantColumnTransformer.evaluateWithSelection(ConstantColumnTransformer.java:46)
  at org.apache.iotdb.calc.transformation.dag.column.AbstractCaseWhenThenColumnTransformer.evaluate(AbstractCaseWhenThenColumnTransformer.java:99)
  at org.apache.iotdb.calc.transformation.dag.column.ColumnTransformer.tryEvaluate(ColumnTransformer.java:46)
  at org.apache.iotdb.calc.execution.operator.process.FilterAndProjectOperator.getTransformedTsBlock(FilterAndProjectOperator.java:273)
  at org.apache.iotdb.calc.execution.operator.process.FilterAndProjectOperator.next(FilterAndProjectOperator.java:133)
  at org.apache.iotdb.calc.execution.operator.Operator.nextWithTimer(Operator.java:48)
  at org.apache.iotdb.db.queryengine.execution.operator.sink.IdentitySinkOperator.next(IdentitySinkOperator.java:109)
  ...

The THEN branch is evaluated with a selection array whose length comes from the WHEN column (2 positions),
while the current input block has only 1 position.

iotdb-case-repro.sql

What did you expect to see?

Expected: 3 rows (d0/hour 0, d1/hour 0, d1/hour 1), each 1.0, 1.0.

What did you see instead?

error code: 301

Anything else?

No response

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Activity

  1. HTHou commented on Oct 4, 2026

    @HTHou
    Contributor
  2. JackieTien97 commented on Oct 6, 2026

    @JackieTien97
    Contributor

    I think this is a bug, and I'll fix it soon

  3. added theissue type on Oct 6, 2026
  4. JackieTien97 commented on Oct 7, 2026

    @JackieTien97
    Contributor

    The root cause is a cache lifecycle bug in the ColumnTransformer DAG used by FilterAndProjectOperator.

    Cached columns are normally released when their reference count reaches zero. However, CASE short-circuit evaluation explicitly clears its child expression caches. If a later projection reuses one of those expressions, recomputation resets its cache reference count to the full number of references in the DAG, without accounting for references already consumed in the current block.

    For example, when an aggregate column is referenced by both the CASE condition and a subsequent output:

    1. The CASE condition consumes the aggregate column, then CASE clears its cache.
    2. The subsequent output recomputes the column, resetting its reference count to 2.
    3. That output consumes one reference, leaving the cached column with a count of 1.
    4. On the next TsBlock, LeafColumnTransformer.initFromTsBlock() updates the input but does not invalidate the cached column. A subsequent tryEvaluate() therefore reuses data from the previous block.

    This mixes columns from different TsBlocks. Different block sizes can cause out-of-bounds access or produce an inconsistent result block, leading to client decoding errors. Equal-sized blocks can also silently return stale values.

    This explains why putting CASE last avoids the reported failure: CASE clears the shared cache after the other output has consumed it. Aggregation and the non-pushdown filter expose the problem by producing multiple input blocks; the underlying defect is in expression cache management.

    A fix should ensure that expression caches cannot survive across input blocks, including caches of intermediate expressions and descendants of nested CASE expressions, while preserving reuse within the current block.

    @ColinLeeo You can take a look and fix that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions