Visitar URL original
[AQE-PR-4] Late-bound materialized routing by wirybeaver · Pull Request #72 · wirybeaver/pinot · GitHub
Skip to content

[AQE-PR-4] Late-bound materialized routing - #72

Draft
wirybeaver wants to merge 1 commit into
pinot-aqe-staged-dispatchfrom
pinot-aqe-late-bound-routing
Draft

wirybeaver wants to merge 1 commit into
pinot-aqe-staged-dispatchfrom
pinot-aqe-late-bound-routing

Conversation

@wirybeaver

@wirybeaver wirybeaver commented Oct 4, 2026 •

Copy link
Copy Markdown
Owner

Summary

Bind completed partition handles to consumers immediately before serialization and dispatch.

producer handles
     |
validate P x M coverage
     |
partition p -> consumer worker p
     |
serialize + dispatch
     |
read assigned local/remote files

Worker counts stay fixed. Runtime partition coalescing is a follow-up, not part of this PR.

Stack

Builds on AQE-PR-3: #71
Roadmap: apache#19485

Validation

Local routing/SerDe tests and real-RPC GROUP BY queries (normal and empty output) passed; style/license checks passed.

Reproduce full-query real-RPC tests

From this PR's checkout, with JDK 25+. The fixture starts its own servers; no external cluster is needed. Two query servers execute real GROUP BY operators, gRPC dispatch and materialized file reads; parallelism stays fixed:

./mvnw -B -ntp -pl pinot-query-runtime -am \
  '-Dtest=QueryRunnerTest#testMaterializedStagedQuery' \
  -Dsurefire.failIfNoSpecifiedTests=false \
  -Dmaven.compiler.showDeprecation=false test

Local report: 2 cases passed:

  • Nonempty GROUP BY ... ORDER BY: schema and rows match the nonmaterialized baseline; late-bound input handles are present.
  • The same query filtered to empty output: the empty result/schema still matches baseline through the complete staged path.

These integrate materialized exchange (PR2) + staged dispatch (PR3) + late binding (PR4). They are also rerun as regression cases in PR5, not counted as new PR5 features.

Test source · Results by phase · Adaptive extension / AQE-PR-5.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant