Parquet V3 reader: PREWHERE with a shared sub-expression fails with NOT_FOUND_COLUMN_IN_BLOCK or returns a wrong count #2604

Describe the bug

With the native Parquet V3 reader (input_format_parquet_use_native_reader_v3 = 1), a PREWHERE with two conjuncts that share a sub-expression can fail with NOT_FOUND_COLUMN_IN_BLOCK (Code: 10). The reader splits the PREWHERE into per-condition steps, and the shared sub-expression becomes a step boundary, but that sub-expression has no column slot in the reader block, so the lookup throws. If the generated name of the shared sub-expression equals the name of a physical column in the file, the lookup binds to the physical column instead, and the query returns a wrong result without any error.

To Reproduce

Each case runs in its own clickhouse local process. The scripts generate their own Parquet file in the working directory, so run them in an empty directory.

Case A (exception):

SET engine_file_truncate_on_insert = 1;

INSERT INTO FUNCTION file($$shared_intermediate.parquet$$, Parquet)
    SELECT $$foo$$ AS a, $$baz$$ AS b, 100 AS ts;
SELECT $$rows written$$, count() FROM file($$shared_intermediate.parquet$$, Parquet);

WITH flat AS
(
    SELECT
        multiIf(a = $$foo$$, $$bar$$, NULL) AS m1,
        multiIf(a = $$foo$$, b, NULL) AS m2,
        ts
    FROM file($$shared_intermediate.parquet$$, Parquet, $$a Nullable(String), b Nullable(String), ts Nullable(Int64)$$)
)
SELECT m2, ts, rn
FROM
(
    SELECT
        f.m2,
        f.ts,
        row_number() OVER (PARTITION BY 1 ORDER BY f.ts DESC) AS rn
    FROM flat f
    INNER JOIN (SELECT $$bar$$ AS k) p ON f.m1 = p.k
    WHERE f.m2 IS NOT NULL
)
WHERE rn = 1
SETTINGS input_format_parquet_use_native_reader_v3 = 1, optimize_move_to_prewhere = 1, enable_analyzer = 1, query_plan_enable_optimizations = 1, query_plan_optimize_prewhere = 1, enable_join_runtime_filters = 1, join_algorithm = $$hash$$, query_plan_join_swap_table = false;

Case B (wrong result):

SET engine_file_truncate_on_insert = 1;

INSERT INTO FUNCTION file($$name_collision.parquet$$, Parquet,
    $$a String, b Int64, c Int64, `equals(length(a), 3_UInt8)` UInt8$$)
    SELECT $$foo$$, 5, 7, 0 FROM numbers(10);
SELECT $$rows written$$, count() FROM file($$name_collision.parquet$$, Parquet, $$a String, b Int64, c Int64, `equals(length(a), 3_UInt8)` UInt8$$);

SELECT count(), max(`equals(length(a), 3_UInt8)`)
FROM file($$name_collision.parquet$$, Parquet, $$a String, b Int64, c Int64, `equals(length(a), 3_UInt8)` UInt8$$)
PREWHERE (length(a) = 3 AND b > 0) AND (length(a) = 3 AND c > 0)
SETTINGS input_format_parquet_use_native_reader_v3 = 1, enable_analyzer = 1, query_plan_optimize_prewhere = 1, optimize_move_to_prewhere = 1;

Run the two cases:

clickhouse local --path ./data-a --multiquery --queries-file repro-case-a.sql
clickhouse local --path ./data-b --multiquery --queries-file repro-case-b.sql

Output on Antalya 26.6.8.20001 (unfixed). The error text is shortened here, and the list of block columns is omitted:

rows written	1
Code: 10. DB::Exception: Not found column equals(__table3.a, ...) in block. There are only columns: ... (NOT_FOUND_COLUMN_IN_BLOCK)

rows written	10
0	0

Expected behavior

Case A returns one row, baz, 100, 1. Case B returns a count of 10 with a max of 0. Output on a local build of Antalya 26.6.8.20001 that includes the change below:

rows written	1
baz	100	1

rows written	10
10	0

Root cause

Reader::preparePrewhere in src/Processors/Formats/Impl/Parquet/Reader.cpp reuses the MergeTree PREWHERE splitter (tryBuildPrewhereSteps) to build per-condition read steps. When two conjuncts share an intermediate node, the splitter makes that node the output of one step and the input of the next. extended_sample_block contains only query outputs and leaf PREWHERE inputs, so the shared node has no slot there, and the cross-step lookup by name throws NOT_FOUND_COLUMN_IN_BLOCK. When the generated name equals a physical column name, as in case B, the lookup resolves to that physical column, so the filter reads the physical values and the query returns a wrong count. The splitter was introduced for the Parquet V3 reader in ClickHouse#93542.

Proposed fix

The excerpt below is the add-step logic from the fix in ClickHouse#107059 for Reader::preparePrewhere. The PR changes Reader.cpp by 50 additions and 9 deletions. A cross-step column must be either an input of the original PREWHERE or an intermediate written to a dedicated prewhere-output slot in extended_sample_block. Otherwise the reader falls back to a single step, and the split is used only when a filter is needed. In the excerpt, ... in prewhere_info_patched = ...; stands for the make_shared of PrewhereInfo built from dag.clone() and filter_column_name (the same statement in its old and its new place), and ... in the for header stands for the loop bound prewhere_expr_info.steps.size().

@@ -789,15 +789,57 @@ void Reader::preparePrewhere()
         if (!actions_settings.has_value())
             actions_settings.emplace();
 
-        auto prewhere_info_patched = ...;
-        prewhere_info_patched->need_filter = needs_filter;
         PrewhereExprInfo prewhere_expr_info;
+        bool success = false;
 
-        bool success = tryBuildPrewhereSteps(
-            prewhere_info_patched,
-            *actions_settings,
-            prewhere_expr_info,
-            /*force_short_circuit_execution*/ false);
+        /// The per-condition split only registers kept prewhere outputs while filtering, so it is
+        /// used only when needs_filter is true; otherwise fall through to the single step below.
+        if (needs_filter)
+        {
+            auto prewhere_info_patched = ...;
+            prewhere_info_patched->need_filter = needs_filter;
+
+            success = tryBuildPrewhereSteps(
+                prewhere_info_patched,
+                *actions_settings,
+                prewhere_expr_info,
+                /*force_short_circuit_execution*/ false);
+
+            /// A cross-step column is addressable only if it is an original prewhere input or an
+            /// intermediate an earlier step wrote to a dedicated prewhere-output slot. Otherwise the
+            /// split step is unaddressable (or would resolve to a physical column that only shares the
+            /// generated name), so fall back to a single step.
+            NameSet addressable_columns;
+            for (const auto & col : dag.getRequiredColumns())
+                addressable_columns.insert(col.name);
+
+            for (const auto & step : prewhere_expr_info.steps)
+            {
+                if (!success)
+                    break;
+                for (const auto & col : step->actions->getActionsDAG().getRequiredColumns())
+                {
+                    if (!addressable_columns.contains(col.name))
+                    {
+                        success = false;
+                        break;
+                    }
+                }
+                if (!success)
+                    break;
+
+                /// An intermediate this step computes can be read by a later step only through a
+                /// dedicated prewhere-output slot: one present in `extended_sample_block` whose slot is
+                /// not an original output column. This mirrors how `add_single_step` registers prewhere
+                /// outputs, and excludes generated names that only collide with physical columns.
+                for (const auto * node : step->actions->getActionsDAG().getOutputs())
+                {
+                    auto idx = extended_sample_block.findPositionByName(node->result_name);
+                    if (idx.has_value() && !sample_block_to_output_columns_idx.at(*idx).has_value())
+                        addressable_columns.insert(node->result_name);
+                }
+            }
+        }
 
         if (success)
         {
@@ -805,8 +847,7 @@ void Reader::preparePrewhere()
             for (size_t i = 0; ...; ++i)
             {
                 auto filter = prewhere_expr_info.steps[i];
-                if (needs_filter)
-                    add_single_step(filter->actions->getActionsDAG(), filter->filter_column_name, true, i);
+                add_single_step(filter->actions->getActionsDAG(), filter->filter_column_name, true, i);
             }
         }

The upstream change also adds regression tests (SQL and reference files under tests/queries/0_stateless), which are not reproduced here.

Key information

• Antalya 26.6.8.20001 and Antalya 26.6.8.20002 (official builds): both cases reproduce (case A raises Code 10, case B returns 0 0).
• Upstream v26.6.8.7-stable: both cases reproduce in the same way.
• Upstream v26.9.21.9-stable: both cases pass, so the fix is in that release line.
• A local build of Antalya 26.6.8.20001 that includes the change above, together with other local changes: both cases pass. The change was not tested alone.
• The repro needs input_format_parquet_use_native_reader_v3 = 1 and enable_analyzer = 1. It reads a local file through file(), with no object storage and no catalog.
• Related symptom in the same code path: with the legacy analyzer (enable_analyzer = 0), a PREWHERE with several conjuncts plus a WHERE raises LOGICAL_ERROR Code 49 ("Unexpected column in sample block") on both official 26.6 builds. The patched build returns rows for those queries.

How this affects us

We run an Iceberg data lake on Antalya 26.6, whose data files are Parquet. We ran both cases on the official 26.6 builds and on a build carrying the upstream fix to check it. We have not measured this in production. The exception form is visible to the user, but the wrong-count form returns no error, so it can pass unnoticed.

Related

• Fix NOT_FOUND_COLUMN_IN_BLOCK in the native Parquet V3 reader prewhere handling ClickHouse/ClickHouse#107059 (merged 2026-07-29): the upstream fix for this defect. The fix commit is 954bf14, which is part of that PR.
• multistage prewhere in parquet reader v3 ClickHouse/ClickHouse#93542 (merged 2026-01-21): introduced the multi-step PREWHERE in the Parquet V3 reader that this defect comes from.
• IS NOT NULL on Iceberg with V3 parquet reader throws NOT_FOUND_COLUMN_IN_BLOCK ClickHouse/ClickHouse#106147 (closed 2026-06-19) and Fix NOT_FOUND_COLUMN_IN_BLOCK for IS NOT NULL on Iceberg/S3 with the Parquet V3 reader ClickHouse/ClickHouse#106443 (merged 2026-06-19): the same error code for a different shape (IS NOT NULL on Iceberg with the V3 reader).
• Altinity tracker: a search for Parquet PREWHERE with NOT_FOUND_COLUMN_IN_BLOCK (all states) found no matching issues.