Proposal: Pluggable SQL statement parsing, planning, and DDL execution #26084

Is your feature request related to a problem or challenge?

I would like downstream applications to register custom top-level SQL statements and use them through DataFusion’s normal session APIs, without maintaining a separate SQL dispatcher or adding a command-specific variant to DataFusion core for each feature.
For example, an extension could implement Snowflake-style CREATE SEMANTIC VIEW. The request is not to implement semantic views in core, but to provide the extension points needed to carry a custom statement from parsing through planning to execution.

Describe the solution you'd like

Provide three complementary extension points:

1. Statement parser: Recognize custom statements and return a typed extension AST, reusing existing tokenization, identifier/expression parsing, and diagnostics. A nonmatching extension must leave the token position unchanged; malformed recognized syntax should report its error rather than silently fall back. Extension precedence should be explicit.
2. Statement planner: Convert the extension AST into a logical plan or typed DDL command. Provide access to normal name resolution, expression planning, type checking, and dependency discovery. Parsing and planning must not execute mutations.
3. DDL executor: Execute custom commands asynchronously through the standard session execution path. Custom DDL must respect restrictions such as with_allow_ddl(false) and execute once per explicit command execution—not per batch or partition. Preparing, describing, or explaining a command must not trigger its mutations.
   These extensions should be registered through parser/session configuration and support an extensible statement representation, rather than requiring a new core enum variant for every downstream command.
   For example, a downstream extension could accept:

CREATE SEMANTIC VIEW analytics.sales
TABLES (
    orders AS public.orders PRIMARY KEY (id)
)
DIMENSIONS (
    orders.country AS orders.country
)
METRICS (
    orders.revenue AS SUM(orders.amount)
);

The intended path is:

SQL → registered parser → extension AST
    → registered planner → logical command
    → registered executor

The semantic implementation would remain downstream. With no extensions registered, existing SQL behavior should remain unchanged.

Describe alternatives you've considered

1. Application-level parser wrapper and dispatcher: Provides a workable integration, but requires each application to maintain routing outside the standard session SQL path.
2. Command-specific changes to core: Solve an individual use case but require further core changes for each additional statement.
3. External-table syntax or side-effecting functions: May suit narrower use cases, but do not provide a general extension point for new statements with a distinct DDL lifecycle.

Additional context

The initial scope is top-level statements—not arbitrary grammar injection, a semantic-view implementation, a new execution engine, or a stable binary plugin ABI.
The architectural inspiration is Calcite-style parser/DDL extensibility, not a requirement to reproduce its parser generator or APIs. Trait names and registration details are open for discussion.
Would maintainers prefer parser/planner extensibility and DDL execution as separate proposals or PRs? Is there an existing mechanism that covers this complete path?