Why AQE? Let runtime data change the remaining plan
SET adaptiveExecution=true;
SET numGroupsLimit=100000;
SELECT col1, COUNT(*) FROM a GROUP BY col1;
Suppose the plan assigns eight workers to the final GROUP BY, but the upstream aggregation produces only a few KiB across eight HASH partitions. Running all eight consumers wastes scheduling and operator state. AQE-PR-5 can use the observed size to coalesce those partitions into one consumer, retaining every original partition without rehashing it.
initial plan: producer group -> 8 HASH partitions -> 8 consumers -> broker
runtime: producers finish -> stats/manifests -> AQE rules / replan
adaptive plan: same 8 partitions -----------------> 1 consumer -> broker
Coalescing is the first rule, not the framework's limit. Ordered rules can call a physical replanner and return a replacement pending plan, changing operators, stages, routing and parallelism—not just a worker count.
Another example: skip work when the build is empty
SELECT /*+ joinOptions(join_strategy='dynamic_broadcast') */ o.order_id
FROM orders o
WHERE o.customer_id IN (
SELECT c.customer_id FROM customers c WHERE c.region = 'new-region'
);
Suppose orders is large, but the customer filter produces zero rows at runtime. The result must be empty: preparing and executing probe-side SSE requests is wasted work. AQE-PR-1 uses that runtime fact to skip those requests, while preserving the normal MSE EOS and statistics path.
Query Execution Flow
The two branches below are complementary, independent execution paths.
SQL -> MSE plan
|
run producer/build work
|
+------------------+----------------------+
| |
dynamic-broadcast build selected HASH exchange
(existing breaker) |
| files + handles + row/byte stats
empty build? [AQE-PR-1] [AQE-PR-2]
/ \ |
yes no all workers succeed + streams close
| | [AQE-PR-3]
skip probe normal probe |
SSE requests SSE execution snapshot -> ordered AQE rules
\ / [AQE-PR-5]
result/EOS / \
coalesce M -> K other rule/replanner*
\ /
validate + publish pending plan
[AQE-PR-5]
|
rebuild ready stage groups
[AQE-PR-3 + AQE-PR-5]
|
bind original partition handles
[AQE-PR-4 + AQE-PR-5]
|
dispatch ready consumer group
[AQE-PR-3]
|
read assigned partition files
[AQE-PR-2 + AQE-PR-4]
|
broker reduction
* Extension point: PR5 ships coalescing, not skew-splitting or join rules. Replanning changes only unsubmitted work; completed stages and the output contract stay fixed. Each new dispatch group must fit the original thread budget, while an individual stage may grow.
Implementation Roadmap
| PR |
Deliverable |
Review |
| AQE-PR-1 |
Skip probe SSE work after an empty dynamic-broadcast build. |
apache/pinot#19470 |
| AQE-PR-2 |
Materialized HASH partitions with runtime row/byte statistics. |
wirybeaver/pinot#70 |
| AQE-PR-3 |
Completion barriers and dependency-ordered dispatch. |
wirybeaver/pinot#71 |
| AQE-PR-4 |
Late-bound partition handles for undispatched consumers. |
wirybeaver/pinot#72 |
| AQE-PR-5 |
Rule-driven runtime replanning, with deterministic partition coalescing (M -> K). |
wirybeaver/pinot#73 |
PR1 is independent. Review the adaptive path as an incremental fork stack:
aqe/base -> AQE-PR-2 -> AQE-PR-3 -> AQE-PR-4 -> AQE-PR-5
Upstream PR2–4 remain temporarily closed in favor of these incremental reviews. PR2–4 alone keep worker counts fixed; PR5 adds opt-in adaptation through adaptiveExecution=true. The coalescing target defaults to 64 MiB (aqeTargetPartitionBytes); 0 disables only that rule. The PoC coalesces eligible unary HASH consumers while respecting group caps and local limits. Skew splitting and join rules remain follow-ups—not framework restrictions.
Why AQE? Let runtime data change the remaining plan
Suppose the plan assigns eight workers to the final GROUP BY, but the upstream aggregation produces only a few KiB across eight HASH partitions. Running all eight consumers wastes scheduling and operator state. AQE-PR-5 can use the observed size to coalesce those partitions into one consumer, retaining every original partition without rehashing it.
Coalescing is the first rule, not the framework's limit. Ordered rules can call a physical replanner and return a replacement pending plan, changing operators, stages, routing and parallelism—not just a worker count.
Another example: skip work when the build is empty
Suppose
ordersis large, but the customer filter produces zero rows at runtime. The result must be empty: preparing and executing probe-side SSE requests is wasted work. AQE-PR-1 uses that runtime fact to skip those requests, while preserving the normal MSE EOS and statistics path.Query Execution Flow
The two branches below are complementary, independent execution paths.
*Extension point: PR5 ships coalescing, not skew-splitting or join rules. Replanning changes only unsubmitted work; completed stages and the output contract stay fixed. Each new dispatch group must fit the original thread budget, while an individual stage may grow.Implementation Roadmap
M -> K).PR1 is independent. Review the adaptive path as an incremental fork stack:
Upstream PR2–4 remain temporarily closed in favor of these incremental reviews. PR2–4 alone keep worker counts fixed; PR5 adds opt-in adaptation through
adaptiveExecution=true. The coalescing target defaults to 64 MiB (aqeTargetPartitionBytes);0disables only that rule. The PoC coalesces eligible unary HASH consumers while respecting group caps and local limits. Skew splitting and join rules remain follow-ups—not framework restrictions.