Visitar URL original
[fix](external) Preserve external binary, UUID and timestamp mappings on master by Gabriel39 · Pull Request #68786 · apache/doris · GitHub
Skip to content

[fix](external) Preserve external binary, UUID and timestamp mappings on master - #68786

Merged
Gabriel39 merged 16 commits into
apache:masterfrom
Gabriel39:fix/external-binary-timestamp-mappings-master
Oct 10, 2026
Merged

Gabriel39 merged 16 commits into
apache:masterfrom
Gabriel39:fix/external-binary-timestamp-mappings-master

Conversation

@Gabriel39

@Gabriel39 Gabriel39 commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Related PR: #68532. Supersedes #68603.

Preserve external binary, UUID, and timestamp semantics across connector schemas, JDBC transport, file readers, and external writes. The tables below list the type mappings changed by this PR, including mappings that previously depended on an opt-in property.

Behavior changes

  • The binary mappings listed below now expose VARBINARY and preserve raw bytes, including invalid UTF-8, embedded NULs, and trailing zeros. File TVFs also apply this rule to nested binary fields. DESCRIBE and query results therefore expose binary types and hexadecimal values instead of decoded text.
  • External logical UUID types map to native Doris UUID, including nested fields, independently of binary mapping options. Native UUID inputs and canonical/compact UUID text can be written to Iceberg UUID columns; invalid text fails explicitly. UUID file reads preserve network byte order and can load directly into native Doris UUID columns.
  • Timestamps representing an instant map to TIMESTAMPTZ; timestamps representing local wall-clock fields map to DATETIMEV2. Instant semantics also apply to MySQL TIMESTAMP and ClickHouse DateTime/DateTime64 without an explicit timezone in the type name. TIMESTAMPTZ preserves the instant, not the source timezone identifier or original offset spelling.
  • Catalog properties enable.mapping.varbinary / enable.mapping.timestamp_tz and file TVF properties enable_mapping_varbinary / enable_mapping_timestamp_tz remain accepted but no longer select the mappings below. Absent, false, and true settings produce the same types. Recursive mappings apply to supported ARRAY/MAP/STRUCT fields.
  • JDBC reads and writes preserve UTC instants and supported fractional precision independently of JVM/session timezone. PostgreSQL infinite/out-of-range timestamptz values become SQL NULL, with nullable metadata propagated even for source NOT NULL columns. MySQL retains its configured zero-date handling. Predicates mixing instant and wall-clock timestamps stay local when remote evaluation would change their meaning.
  • Binary partition values retain byte identity and distinguish SQL NULL from literal text. Iceberg UUID writes preserve UUID semantics through ARRAY/MAP/STRUCT leaves, including map keys and values; fixed-width binary writes validate the actual byte count. Paimon legacy ORC routing uses the split's owning branch and historical schema so instant columns use the appropriate reader.
  • MySQL query TVF wrapping distinguishes --1 arithmetic from comments and honors the remote connection’s NO_BACKSLASH_ESCAPES mode. SQL Server UUID range predicates stay local because its GUID ordering differs from Doris. Iceberg UUID value predicates also stay local because file bounds can otherwise prune matching UUID rows.
  • Ordinary views retain binary types and declared VARBINARY bounds. Native-table materialization remains subject to Doris storage restrictions and needs explicit conversion where the target cannot store VARBINARY. Unsupported VARBINARY comparisons/functions remain unsupported; FE validation prevents implicit text coercion for the guarded operations. CDC JSON ingestion retains its existing transport conversions.

Changed read mappings: external system → Doris

“Before” describes the previous default (mapping options absent/false). Where an opt-in mapping already existed, this PR makes that mapping unconditional. p denotes the effective fractional precision supported by the mapper, capped at 6 (HMS uses the catalog timestamp precision); fixed precisions are shown explicitly. n denotes the supported declared byte bound; unqualified VARBINARY uses Doris's default maximum bound. Unchanged source-type mappings are omitted.

External system / access path External type Before After
Hive / HMS BINARY STRING VARBINARY
Hive / HMS TIMESTAMP WITH LOCAL TIME ZONE DATETIMEV2(p) TIMESTAMPTZ(p)
Iceberg binary STRING VARBINARY
Iceberg fixed(n) CHAR(n) VARBINARY(n)
Iceberg uuid STRING UUID
Iceberg timestamptz (shouldAdjustToUTC=true) DATETIMEV2(6) TIMESTAMPTZ(6)
Paimon BINARY(n), VARBINARY(n) / BYTES STRING VARBINARY(n) / VARBINARY
Paimon TIMESTAMP WITH LOCAL TIME ZONE / TIMESTAMP_LTZ(p) DATETIMEV2(p) TIMESTAMPTZ(p)
Fluss BINARY(n), BYTES STRING VARBINARY(n), VARBINARY respectively
Fluss TIMESTAMP_LTZ(p) DATETIMEV2(p) TIMESTAMPTZ(p)
Hudi / Avro logical schema Logical uuid on string STRING UUID
Hudi / Avro logical schema timestamp-millis, timestamp-micros DATETIMEV2(3), DATETIMEV2(6) TIMESTAMPTZ(3), TIMESTAMPTZ(6) respectively
Hudi / Avro logical schema local-timestamp-millis, local-timestamp-micros BIGINT DATETIMEV2(3), DATETIMEV2(6) respectively
MaxCompute TIMESTAMP DATETIMEV2(6) TIMESTAMPTZ(6)
Trino Connector UUID Unsupported by the type mapper UUID
Trino Connector VARBINARY STRING VARBINARY
Trino Connector TIMESTAMP(p) WITH TIME ZONE DATETIMEV2(p) TIMESTAMPTZ(p)
MySQL / MariaDB JDBC BINARY, VARBINARY, TINYBLOB, BLOB, MEDIUMBLOB, LONGBLOB STRING VARBINARY with the supported JDBC-reported byte bound
MySQL / MariaDB JDBC TIMESTAMP(p) DATETIMEV2(p) TIMESTAMPTZ(p)
PostgreSQL JDBC uuid, including array elements STRING UUID
PostgreSQL JDBC bytea STRING VARBINARY with the supported JDBC-reported byte bound
PostgreSQL JDBC timestamptz DATETIMEV2(p) TIMESTAMPTZ(p)
Oracle JDBC BLOB STRING VARBINARY with the supported JDBC-reported byte bound
Oracle JDBC TIMESTAMP(p) WITH LOCAL TIME ZONE DATETIMEV2(p) TIMESTAMPTZ(p)
Oracle JDBC TIMESTAMP(p) WITH TIME ZONE STRING in the connector; unsupported in the legacy client TIMESTAMPTZ(p)
SQL Server JDBC uniqueidentifier STRING UUID
SQL Server JDBC binary, varbinary, image STRING VARBINARY with the supported JDBC-reported byte bound
SQL Server JDBC datetimeoffset(p) STRING TIMESTAMPTZ(p)
DB2 JDBC BLOB, BINARY, VARBINARY STRING VARBINARY with the supported JDBC-reported byte bound
GBase JDBC JDBC BINARY, VARBINARY, LONGVARBINARY, BLOB STRING VARBINARY with the supported JDBC-reported byte bound
ClickHouse JDBC DateTime, DateTime('zone') DATETIMEV2(0) TIMESTAMPTZ(0)
ClickHouse JDBC DateTime64(p[, 'zone']) DATETIMEV2(p) TIMESTAMPTZ(p)
Trino / Presto JDBC uuid, including array elements Unsupported by the type mapper UUID
Trino / Presto JDBC TIMESTAMP(p) WITH TIME ZONE, TIMESTAMP WITH TIME ZONE DATETIMEV2 / STRING, or a metadata precision-parsing failure depending on spelling/path TIMESTAMPTZ(p); precision defaults to 6 when absent
Trino / Presto JDBC connector Unparameterized TIMESTAMP metadata STRING DATETIMEV2(6)
Parquet file TVFs Raw BYTE_ARRAY, including binary fallback for legacy ENUM/BSON annotations STRING VARBINARY
Parquet file TVFs UUID logical annotation on FIXED_LEN_BYTE_ARRAY(16) STRING during legacy schema inference; scanner V2 already supported native UUID; opt-in binary mapping exposed VARBINARY(16) UUID consistently
Parquet file TVFs Logical TIMESTAMP with isAdjustedToUTC=true DATETIMEV2(3/6) TIMESTAMPTZ(3/6) for millis / micros or nanos
ORC file TVFs BINARY STRING VARBINARY
ORC file TVFs BINARY with iceberg.binary-type=UUID STRING UUID
ORC file TVFs TIMESTAMP_INSTANT DATETIMEV2(6) TIMESTAMPTZ(6)

OceanBase JDBC delegates to the MySQL or Oracle mapper according to its database mode and inherits the corresponding changed mappings above.

The file TVF rows apply to schema inference through supported file sources such as S3, HDFS, LOCAL, and HTTP. Existing unzoned timestamp mappings (for example MySQL DATETIME, PostgreSQL timestamp, Iceberg timestamp, and Paimon/Fluss TIMESTAMP) remain DATETIMEV2 and are omitted from the table. Precision beyond microseconds is truncated. Physical binary storage with a recognized STRING/UTF8/DECIMAL annotation retains that logical type.

Changed external schema mappings: Doris → external system

External target Doris type Before After
Iceberg UUID, including nested fields Unsupported by the reverse schema mapper Iceberg uuid
Hive / HMS VARBINARY Unsupported by the reverse schema mapper Hive binary, including nested fields
Paimon DATETIMEV2(p) Paimon TIMESTAMP(6); source precision discarded Paimon TIMESTAMP(p)
Paimon TIMESTAMPTZ(p) Unsupported by the reverse schema mapper Paimon TIMESTAMP_LTZ(p)
MaxCompute TIMESTAMPTZ Unsupported by the reverse schema mapper MaxCompute TIMESTAMP, with UTC microsecond write transport

Release note

The external binary, UUID, and timestamp mappings listed above are determined by source semantics. The former mapping options no longer opt out of VARBINARY or TIMESTAMPTZ. Applications reading these columns may observe changed schemas, binary result formatting, and timestamp semantics. External UUIDs retain native UUID semantics rather than mapping to string or binary.

Validation

  • FE and Java plugin package build passed with 339 targeted unit tests, plus 59 JDBC planning/binding tests. FE Checkstyle passed.
  • 202 BE unit tests passed, covering UUID byte order and nullability, Parquet/ORC data and equality-delete combinations, missing-column defaults, ORC writes, and UUID partitions. Changed C++ lines passed clang-tidy; clang-format 16 and build hygiene checks passed.
  • 13 local regression suites passed: seven UUID file suites, PostgreSQL native UUID and all-types suites, MySQL query TVFs under both SQL modes, Iceberg nested UUID writes, cross-format reads/deletes, and initial defaults. Result files are generated by actual execution and compared in a separate run.
  • Iceberg write coverage includes canonical/compact text, native UUID INSERT SELECT, identity partitions, ARRAY/MAP/STRUCT nesting, NULL and empty containers, invalid inputs, and volatile map expressions whose key/value pairing must be retained. A non-pushable reference verifies UUID equality predicates do not lose rows to file pruning.
  • SQL Server, Trino, and Hudi UUID paths have unit coverage; their external integration suites were not run in this update.
  • Previous revision validation also covered the affected HDFS, Paimon, and HTTP TVF suites.

Check List (For Author)

  • Unit and regression coverage included.
  • User-visible behavior and changed type mappings described.

Check List (For Reviewer who merge this PR)

  • Confirm the release note.
  • Confirm test cases.
  • Confirm document.
  • Add branch pick label if needed.

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review complete at head 01eb76566c253ae33b1d76a459d9ad01c28ddea8 across all 217 changed paths. I request changes for the four inline findings: two P1 correctness/test blockers and two P2 query/reader-selection defects. There were no existing inline comments to carry forward.

Critical checkpoint conclusions:

  • Goal and tests: The change makes external VARBINARY and TIMESTAMPTZ mappings consistent across connectors and adds unit and regression coverage. The new Iceberg wire test cannot exercise its intended NULL marker (P1); the Java JDBC bind tests inject dialect names directly and miss the BE parameter mismatch (P1).
  • Scope: The large connector, FE, BE, and golden-result surface follows the mapping change. No unrelated change was substantiated during the file sweep.
  • Concurrency: Catalog marker migration runs after fenced replay under the catalog manager write lock; journal-before-local-replay ordering was traced. No new thread-safety or lock-order defect was substantiated.
  • Lifecycle: Catalog promotion/rebuild, JNI scanner and writer selection, Paimon native/JNI dispatch, and Iceberg partition write/commit lifecycles were traced. No separate ownership, cleanup, or static-initialization defect was found.
  • Configuration: Catalog mapping markers are migrated and enforced as true; the legacy ORC LTZ table option is checked during scan planning. The latter check is too broad for unprojected LTZ fields (P2). No new runtime-updatable setting was identified.
  • Compatibility: Production Iceberg FE uses Thrift field 20 for NULL keys, while the new manual test writes field 19 (P1). The old-FE/new-BE Iceberg FIXED-write gap is explicitly documented as an accepted rolling-upgrade limitation, so it is not reposted.
  • Parallel paths: Static/dynamic Iceberg partitions, JDBC table scans/TVFs and writes, and Paimon native/JNI reads were compared. The TVF wrapper breaks a valid terminal line comment (P2); the JDBC write path sends a numeric dialect where its Java factory expects a name (P1).
  • Conditions: Binary and timestamp type guards and historical-schema checks were reviewed. The Paimon full-schema LTZ condition forces JNI even without an LTZ read, and can reject a metadata-column query (P2).
  • Coverage and results: BE, FE, Java, and regression tests plus 44 changed golden files were inspected. Expected types/values align with the new mappings apart from the findings above. No build or test was executed because this review's instructions require static inspection; author/CI claims were not treated as independent validation.
  • Observability: The inspected error paths report failures through existing statuses/exceptions; no separate missing log or metric was substantiated.
  • Persistence and failover: Catalog marker edit-log writes, replay, and master promotion were traced; no unjournaled state or distinct failover bug was established.
  • Writes and atomicity: Iceberg Arrow/partition conversion and JDBC bind/transaction paths were reviewed. The JDBC handler-selection error affects TIMESTAMPTZ write semantics (P1); no separate commit or crash-safety issue was substantiated.
  • FE/BE transport: The new Iceberg Thrift field is set correctly in production, while its test fixture is wrong; the JDBC enum reaches Java as the wrong string (both inline). Other changed carriers were traced without a separate issue.
  • Performance and other: Unneeded Paimon JNI fallback loses native ORC splitting and parallelism (P2). No further evidenced performance or correctness issue survived the final sweep.

User focus: no additional focus was provided. All four new issues are inline; no existing blocking comment IDs apply.

Comment thread be/test/exec/sink/writer/iceberg/viceberg_table_writer_test.cpp Outdated
Comment thread be/src/exec/sink/writer/vjdbc_table_writer.cpp
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29173 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 01eb76566c253ae33b1d76a459d9ad01c28ddea8, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17425	5622	5556	5556
q2	2077	298	290	290
q3	10132	1411	853	853
q4	4680	474	350	350
q5	7443	797	533	533
q6	190	190	158	158
q7	773	804	609	609
q8	9254	1244	1148	1148
q9	5836	4484	4442	4442
q10	6825	1313	1038	1038
q11	451	259	232	232
q12	638	401	280	280
q13	18077	3083	2372	2372
q14	277	265	248	248
q15	q16	738	725	660	660
q17	1220	763	587	587
q18	7107	6243	6199	6199
q19	1096	951	608	608
q20	383	349	228	228
q21	5671	2511	2613	2511
q22	385	346	271	271
Total cold run time: 100678 ms
Total hot run time: 29173 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	6538	6485	6433	6433
q2	696	592	596	592
q3	5653	5676	5214	5214
q4	2083	2392	1595	1595
q5	5807	5692	5736	5692
q6	259	192	161	161
q7	2365	2002	1836	1836
q8	3103	2749	2827	2749
q9	8459	8025	7746	7746
q10	3645	3597	3251	3251
q11	608	416	399	399
q12	666	713	503	503
q13	2756	3074	2361	2361
q14	288	295	264	264
q15	q16	664	691	617	617
q17	7792	6969	6901	6901
q18	13178	12223	13172	12223
q19	949	810	785	785
q20	2179	2171	1951	1951
q21	5847	4697	4772	4697
q22	492	467	415	415
Total cold run time: 74027 ms
Total hot run time: 66385 ms

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes. Static review of PR #68786 at f2c7b1d41ad2ba9d5ab12e2ee3b6859123ba5563 across all 219 changed files. Ten new inline findings: two P1, six P2, and two P3. The P1s are unreadable Fluss binary-partition tables after catalog migration and a MySQL mixed-timestamp predicate that can silently discard rows. The remaining findings cover Iceberg UUID partition reads, migration lock scope, PrestoDB and old Oracle zoned writes, Oracle/Trino year bounds, and two regression suites.

Convergence and prior threads. Three normal full-review passes and separate risk-focused passes covered the changed paths and their upstream/downstream carriers. Round three found three new valuable candidates; the review is incomplete under the review contract's three-round convergence limit, although every currently identified candidate has been verified and accepted or dismissed and every accepted issue is inline below. The four existing inline threads were rechecked at this head: the Iceberg Thrift NULL-key fixture and JDBC terminal-comment/Paimon fallback cases are fixed; the JDBC dialect-name P1 does not apply because the generated Thrift overload returns names and its new test covers the parameter. No existing P0/P1 comment remains applicable, so there are no carried blocker IDs. The related PR #68532 and this PR already acknowledge the mixed-version FE schema-serving window; it is recorded as a known limitation rather than a duplicate inline issue. No additional user focus was provided.

Critical checkpoint conclusions (code-review skill Part 1.3):

  • Goal and proof: The change enables default VARBINARY and zoned-timestamp mappings across external catalogs and read/write transports. It adds broad unit, integration and regression cases, but the ten inline gaps mean the goal is not fully met; the tests were inspected, not run here.
  • Scope and clarity: The 219-file change is broad but follows the shared mapping rollout. The two new regression blocks violate repository output/cleanup conventions; no unrelated production change was established.
  • Concurrency and locks: Master promotion holds the outer CatalogMgr write lock while a deferred authorization-plugin close runs (M-002). No further new lock-order failure was substantiated. Existing connector-close work under the lock also occurs in ordinary ALTER and is a separate pre-existing pattern.
  • Lifecycle and initialization: CREATE, ALTER, replay, migration, connector reset and promotion were traced. Journal-first migration and retry/checkpoint ordering are coherent; the authorization cleanup lock scope is not. No new cross-translation-unit static initializer dependency was found.
  • Configuration: The binary and timestamp mapping markers are compatibility settings forced or normalized to true, rather than dynamically switchable options. That makes Fluss's unsupported binary partition mode unavoidable after migration or ALTER (M-005).
  • Compatibility: FE/BE Thrift field 20 and JDBC dialect transport were rechecked; the old fixture and enum-name concerns are resolved. The acknowledged mixed-FE-version schema window remains. The newly supported zoned Oracle write path lacks an ojdbc6 bind (M-011).
  • Parallel paths: Native/JNI, scalar/array, table/TVF, static/dynamic Iceberg writes, partition constants, NULL, predicates and external sinks were traced. M-001, M-004, M-007, M-008, M-010 and M-011 are the concrete mismatches; related carriers examined did not yield another distinct issue.
  • Conditional checks: Paimon historical ORC field-ID routing and binary-expression guards were checked. JDBC's new instant guard misses the MySQL two-column, mixed-type comparison (M-010); the out-of-range Oracle/Trino readers lack the equivalent PostgreSQL bound check (M-007/M-008).
  • Test coverage and negative cases: Added cases cover many ordinary bytes, time zones and NULLs, but omit a UUID identity scan, upgraded Fluss binary partitions, official PrestoDB and old Oracle writes, Oracle/Trino out-of-range values, and a non-UTC MySQL two-column comparison. M-003/M-009 also need runner-generated ordered output and no post-test table drop.
  • Test results: Golden-output changes were inspected against the code paths; no additional wrong expected row was substantiated. No build, unit test, regression test or live database case was executed in this review, so author/CI claims are not independent validation.
  • Observability: Existing catalog/connector error paths and identifiers were inspected; no separate missing log or metric was substantiated. Explicit range rejection would make M-007/M-008 diagnosable instead of packing unsupported years.
  • Persistence and transactions: Catalog marker logs are written before local apply, replay reproduces the marker, and crash/retry/checkpoint ordering appears coherent. No new storage visible-version, delete-bitmap or transaction-log inconsistency was found.
  • Writes and crashes: Iceberg static/dynamic partition routing, typed commit values and file cleanup were traced without another proven atomicity or crash leak. The concrete new write failures are M-004 and M-011; the lock issue is M-002.
  • FE-to-BE values: The new Thrift NULL-key field and JDBC dialect/parameter transport were checked at each sender/reader reached by the diff; no additional missing propagation was found.
  • Performance: Paimon schema checks are cached by scan context/schema ID, and no distinct unbounded allocation or hot-loop regression was proven. Slow plugin cleanup under the global catalog lock is the concrete contention issue (M-002).
  • Other issues: The remaining speculative or already-reported candidates were dismissed with code evidence or existing review context. The ten comments below are the complete accepted set from this capped static review.

Comment thread fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java Outdated
Comment thread regression-test/suites/datatype_p0/test_varbinary_sql_support.groovy Outdated
@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 35956 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f2c7b1d41ad2ba9d5ab12e2ee3b6859123ba5563, data reload: false

------ Round 1 ----------------------------------
region	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:24	NULL	utf-8	NULL	NULL	
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17621	5598	5545	5545
q2	2094	258	224	224
q3	10263	1467	832	832
q4	4707	539	386	386
q5	q6	888	275	263	263
q7	q8	22025	1753	1546	1546
q9	9349	8731	8739	8731
q10	8340	3684	3191	3191
q11	460	268	275	268
q12	714	472	323	323
q13	17815	4477	3702	3702
q14	430	423	392	392
q15	q16	514	499	446	446
q17	1244	783	623	623
q18	6962	6126	6056	6056
q19	1141	989	645	645
q20	428	366	249	249
q21	5605	2269	2303	2269
q22	387	337	265	265
Total cold run time: 110987 ms
Total hot run time: 35956 ms

----- Round 2, with runtime_filter_mode=off -----
region	Doris	NULL	NULL	5	240	1201	NULL	147	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:24	NULL	utf-8	NULL	NULL	
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	6182	6142	6154	6142
q2	927	700	709	700
q3	3013	3200	2679	2679
q4	2048	2162	1569	1569
q5	q6	2373	214	163	163
q7	q8	39388	3370	2985	2985
q9	17697	16767	16764	16764
q10	9320	4899	4491	4491
q11	634	436	406	406
q12	1024	749	522	522
q13	5004	4462	3696	3696
q14	400	403	368	368
q15	q16	478	498	448	448
q17	13096	12010	11694	11694
q18	7948	7509	7418	7418
q19	940	831	892	831
q20	2235	2212	1982	1982
q21	15136	14385	14292	14292
q22	522	472	400	400
Total cold run time: 128365 ms
Total hot run time: 77550 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 25.44 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f2c7b1d41ad2ba9d5ab12e2ee3b6859123ba5563, data reload: false

query1	0.00	0.00	0.01
query2	0.15	0.09	0.08
query3	0.37	0.21	0.21
query4	1.60	0.21	0.21
query5	0.32	0.30	0.30
query6	1.16	0.66	0.65
query7	0.05	0.01	0.00
query8	0.10	0.07	0.06
query9	0.51	0.39	0.40
query10	0.61	0.62	0.61
query11	0.33	0.17	0.18
query12	0.32	0.19	0.18
query13	0.51	0.52	0.51
query14	0.89	0.90	0.90
query15	0.67	0.58	0.59
query16	0.37	0.38	0.38
query17	1.06	1.02	1.05
query18	0.33	0.30	0.29
query19	2.01	1.88	1.86
query20	0.02	0.01	0.01
query21	15.40	0.34	0.31
query22	4.82	0.12	0.12
query23	15.85	0.49	0.28
query24	2.34	0.55	0.42
query25	0.15	0.10	0.10
query26	0.71	0.27	0.21
query27	0.10	0.10	0.09
query28	3.48	0.91	0.44
query29	12.51	4.25	3.29
query30	0.37	0.22	0.23
query31	2.76	0.62	0.35
query32	3.23	0.63	0.51
query33	2.98	2.88	3.04
query34	15.85	4.12	3.42
query35	3.40	3.37	3.40
query36	0.59	0.50	0.49
query37	0.12	0.10	0.09
query38	0.07	0.07	0.07
query39	0.06	0.06	0.06
query40	0.18	0.16	0.15
query41	0.11	0.06	0.05
query42	0.06	0.06	0.05
query43	0.06	0.06	0.06
Total cold run time: 96.58 s
Total hot run time: 25.44 s

Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Oct 8, 2026
### What problem does this PR solve?

Related PR: apache#68786

Problem Summary: Preserve unconditional Fluss binary and timestamp types and encode binary/UUID partition constants for VARBINARY scans. Keep mixed instant and wall-clock predicates local, release the outer catalog migration lock before plugin cleanup, and reject Oracle/Trino UTC years outside the supported range. Use server-side casts for PrestoDB and legacy Oracle zoned writes, and avoid PrestoDB's unsupported typed timestamp getter. Update stale SQL Server unit-test expectations and generate ordered regression outputs through actual execution.

### Release note

Fluss BINARY/BYTES always map to VARBINARY, TIMESTAMP_LTZ to TIMESTAMPTZ, and TIMESTAMP to DATETIMEV2. PrestoDB and older Oracle JDBC drivers support zoned timestamp writes through explicit server conversions.

### Check List (For Author)

- Test: 282 targeted FE/connector/JDBC tests passed, including real PrestoDB and Oracle 11g round trips in three JVM zones. Both VARBINARY and PostgreSQL regression suites generated their outputs and passed comparison reruns. Full FE Checkstyle and clang-format 16 passed.
- Behavior changed: Yes, unconditional Fluss mappings and corrected external partition/time semantics.
- Does this need documentation: No, follows the external type mapping behavior documented by the parent change.
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review complete at head 542257b6961afcc5d7c93c4bd170dc82f0320a05 after two convergence rounds and a final sweep of all 239 changed paths. Request changes: the new MySQL timestamp predicate case can silently discard rows. The three inline comments cover that case, Iceberg binary partition metadata, and regression test standards. No additional user review focus was supplied. This was static inspection; no build or test was run.

Code-review checkpoints:

  1. Goal and tests: The change wires binary and zoned timestamp mappings through FE, JNI, BE, catalog migration, and Iceberg writes, with new unit and regression source. The reported JDBC and Iceberg paths prevent complete correctness, and their specific cases lack coverage.
  2. Scope and clarity: The broad change follows the required cross-layer type contracts. No unrelated production change was substantiated as a separate issue.
  3. Concurrency and locks: Master promotion journals and applies marker updates under the catalog write lock. Authorization cleanup is deferred until unlock, while connector close can still perform I/O under that lock; this residual is within existing P2 thread 4218883248. No new lock-order or deadlock path was established.
  4. Lifecycle: Promotion gates serving, replays the journal, migrates markers, then resumes checkpoint/service work. Reset and connector rebuild paths were traced; no distinct retry or startup divergence was found.
  5. Configuration: CREATE, ALTER, and migration normalize the two mapping markers to true. This is an intentional one-way compatibility migration; no dynamic toggle is promised.
  6. Compatibility: The static-null Thrift set uses field 20 and FE/BE agree on its representation. Supported rolling upgrades put BE before FE; acknowledged mixed-version limitations remain. No external file rewrite is introduced.
  7. Parallel paths and conditions: Legacy and plugin JDBC, CDC/TVF exceptions, external connector reads, and Iceberg static/dynamic writes were compared. The JDBC guard handles wall-clock columns but misses literals; Iceberg's partition-key builder still lacks VARBINARY.
  8. Coverage and results: Relevant tests and generated output changes were inspected, never executed here. New regression checks listed in the P3 comment lack generated output; the reported JDBC literal and Iceberg identity-partition cases are not covered.
  9. Observability and errors: FE warns when partition-item creation fails but then reports the table unpartitioned, obscuring the loss of pruning. Other error/status and BE null/constant handling examined did not yield a distinct new finding.
  10. Persistence and failover: Catalog ALTER logging precedes application, and follower replay/retry use the persisted markers. No new EditLog or checkpoint inconsistency was established.
  11. Writes and crash paths: Iceberg NULL, binary, UUID, timestamp, full-static, hybrid, overwrite, transform, and commit transports were traced. No separate atomicity or crash-leak defect was substantiated.
  12. FE/BE transport: New binary/instant carriers and the Thrift NULL marker were checked on both sides; the older UUID scan-transport thread is addressed.
  13. Performance, memory, and other paths: The Iceberg finding disables FE partition pruning. Fluss still loses FE binary partition metadata/pruning, but its earlier P1 listing/SELECT outage is fixed and the narrower residual belongs to existing thread 4218883211. No distinct memory-accounting or further performance issue survived recheck.

Existing P1 threads 4217826031, 4217826038, 4218883211, and 4218883225 were independently checked; none remains applicable at its reported P1 failure mode, so no existing blocker ID is carried.

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 77.84% (2079/2671)
Line Coverage 66.02% (38158/57795)
Region Coverage 53.57% (35842/66907)
Branch Coverage 57.00% (11527/20223)

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29057 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 542257b6961afcc5d7c93c4bd170dc82f0320a05, data reload: false

------ Round 1 ----------------------------------
============================================
q1	16878	5658	5513	5513
q2	2088	324	248	248
q3	10082	1381	830	830
q4	4645	471	348	348
q5	7467	784	539	539
q6	194	192	157	157
q7	752	791	607	607
q8	9290	1320	1117	1117
q9	5675	4412	4436	4412
q10	6813	1313	1023	1023
q11	443	255	231	231
q12	634	409	285	285
q13	18031	3036	2364	2364
q14	277	276	247	247
q15	q16	734	720	680	680
q17	1174	746	570	570
q18	7084	6201	6176	6176
q19	1098	954	607	607
q20	376	345	225	225
q21	5561	2572	2581	2572
q22	408	327	306	306
Total cold run time: 99704 ms
Total hot run time: 29057 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	6438	6473	6398	6398
q2	700	606	579	579
q3	5583	5638	5293	5293
q4	2002	2153	1538	1538
q5	6103	5653	5743	5653
q6	248	198	157	157
q7	2283	2058	1877	1877
q8	3121	2755	2733	2733
q9	8376	8222	7768	7768
q10	3627	3580	3223	3223
q11	588	413	380	380
q12	662	695	511	511
q13	2724	3039	2375	2375
q14	293	291	272	272
q15	q16	672	679	620	620
q17	7807	7077	6875	6875
q18	13133	12284	12972	12284
q19	927	789	781	781
q20	2186	2188	1951	1951
q21	5702	4665	4752	4665
q22	522	477	411	411
Total cold run time: 73697 ms
Total hot run time: 66344 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 151464 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 542257b6961afcc5d7c93c4bd170dc82f0320a05, data reload: false

query5	4313	573	462	462
query6	413	206	194	194
query7	4808	451	222	222
query8	311	184	167	167
query9	8712	3919	3943	3919
query10	486	310	254	254
query11	5929	3433	3176	3176
query12	140	87	87	87
query13	1254	442	319	319
query14	6553	4742	4437	4437
query14_1	4215	4227	4223	4223
query15	204	204	184	184
query16	941	444	409	409
query17	849	654	547	547
query18	2408	430	317	317
query19	199	169	133	133
query20	81	81	78	78
query21	206	130	112	112
query22	12944	12912	12801	12801
query23	13035	12617	12215	12215
query23_1	12200	12172	12209	12172
query24	6963	821	464	464
query24_1	474	475	470	470
query25	540	418	365	365
query26	1259	257	143	143
query27	2769	436	281	281
query28	4536	1905	1909	1905
query29	1540	600	456	456
query30	288	219	181	181
query31	898	761	647	647
query32	138	93	90	90
query33	507	305	250	250
query34	923	877	496	496
query35	734	756	651	651
query36	808	798	740	740
query37	125	98	90	90
query38	1785	1770	1678	1678
query39	716	684	693	684
query39_1	693	656	672	656
query40	210	112	92	92
query41	66	62	66	62
query42	87	82	85	82
query43	351	374	325	325
query44	1301	678	694	678
query45	180	180	164	164
query46	831	947	557	557
query47	2995	2989	2862	2862
query48	295	300	214	214
query49	562	391	311	311
query50	671	271	207	207
query51	10493	10312	10487	10312
query52	85	83	70	70
query53	187	204	152	152
query54	214	188	176	176
query55	72	74	63	63
query56	217	221	198	198
query57	1600	1577	1522	1522
query58	276	255	245	245
query59	2340	2359	2098	2098
query60	285	257	219	219
query61	149	146	144	144
query62	399	345	284	284
query63	194	161	156	156
query64	2679	930	761	761
query65	3422	3355	3417	3355
query66	1769	409	297	297
query67	20202	20183	20113	20113
query68	3034	999	597	597
query69	371	288	252	252
query70	910	820	805	805
query71	285	224	201	201
query72	2681	2549	2267	2267
query73	518	540	291	291
query74	4567	4470	4289	4289
query75	2328	2283	1905	1905
query76	2187	1031	611	611
query77	354	396	297	297
query78	9114	9008	8510	8510
query79	973	838	508	508
query80	1188	421	324	324
query81	572	324	269	269
query82	544	131	108	108
query83	291	191	174	174
query84	314	112	93	93
query85	844	414	364	364
query86	389	236	229	229
query87	2016	1939	1816	1816
query88	3559	2664	2644	2644
query89	342	290	260	260
query90	1837	180	179	179
query91	153	148	122	122
query92	96	83	86	83
query93	975	991	559	559
query94	623	307	267	267
query95	574	351	389	351
query96	654	518	237	237
query97	2462	2433	2362	2362
query98	156	146	144	144
query99	779	763	655	655
Total cold run time: 233413 ms
Total hot run time: 151464 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 25.49 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 542257b6961afcc5d7c93c4bd170dc82f0320a05, data reload: false

query1	0.01	0.00	0.01
query2	0.15	0.08	0.09
query3	0.37	0.21	0.21
query4	1.62	0.20	0.22
query5	0.32	0.28	0.28
query6	1.16	0.66	0.64
query7	0.04	0.00	0.00
query8	0.08	0.07	0.07
query9	0.48	0.39	0.39
query10	0.56	0.56	0.56
query11	0.32	0.18	0.18
query12	0.31	0.18	0.18
query13	0.51	0.51	0.52
query14	0.90	0.88	0.88
query15	0.68	0.58	0.59
query16	0.35	0.36	0.37
query17	1.06	0.96	1.00
query18	0.33	0.30	0.29
query19	2.04	1.86	1.91
query20	0.02	0.01	0.01
query21	15.41	0.36	0.31
query22	4.81	0.14	0.13
query23	15.85	0.51	0.31
query24	2.33	0.59	0.41
query25	0.14	0.10	0.09
query26	0.74	0.27	0.21
query27	0.11	0.09	0.10
query28	3.46	0.80	0.41
query29	12.47	4.30	3.33
query30	0.37	0.22	0.24
query31	2.76	0.59	0.35
query32	3.22	0.63	0.51
query33	3.04	3.03	3.14
query34	15.81	4.10	3.48
query35	3.36	3.35	3.38
query36	0.58	0.51	0.50
query37	0.12	0.09	0.10
query38	0.08	0.06	0.06
query39	0.06	0.06	0.06
query40	0.18	0.17	0.16
query41	0.11	0.05	0.04
query42	0.06	0.05	0.06
query43	0.06	0.06	0.06
Total cold run time: 96.44 s
Total hot run time: 25.49 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 59.66% (71/119) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 74.79% (89/119) 🎉
Increment coverage report
Complete coverage report

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review status: INCOMPLETE under the requested three-round convergence rule: the final risk pass found a new, independently validated Hive ORC V1 issue (M-9), so another no-new-findings round could not be run. All 265 changed paths were covered in each normal pass, every raised candidate has a disposition, and all eight accepted findings are inline below. Two are P1 (Iceberg partition/file identity and official PrestoDB zoned-array scans); six are P2 (four external JNI timestamp range paths, nested MaxCompute writes, and Hive ORC V1 range packing). I recommend changes before merge.

Existing P1 disposition: no previously reported P1 still applies with its original scope at this head, so existing_blocking_comment_ids is empty. The BE Thrift test uses field 20; the JDBC dialect name resolves the generated Thrift to_string overload; upgraded Fluss nodes decode binary partition names (the conditional old-follower failover case is deduplicated against that thread at lower severity); and both JDBC mixed-timezone predicate forms are held local by hasWallClockValue with remote LIMIT suppressed when a filter is dropped. Existing P2/P3 inline threads remain duplicate fences and were not reposted. The user focus file listed no additional focus.

Code-review critical checkpoints:

  1. Goal and tests: the change aims to preserve external binary bytes and zoned instants across read/write paths. Ordinary, NULL, DST, pre-epoch, partition, and nested fixtures were added, but the eight findings show the goal is incomplete and their boundary/mixed-value cases lack coverage.
  2. Scope: the 265-path change is broad but centered on type mapping, transport, catalog migration, and tests; no unrelated production edit was identified.
  3. Concurrency: catalog migration journals/applies under the global catalog write lock and performs detached cleanup after unlock; writer caches are per sink instance. The Iceberg path-key collision is a correctness fault, not a lock race; no separate lock-order or shared-state defect survived.
  4. Lifecycle/statics: connector reset/close and master-promotion sequencing were traced; function-local epoch statics have ordered initialization. No new cross-translation-unit initializer dependency or unreleased lifecycle was found.
  5. Configuration: legacy binary/instant mapping flags now normalize to true on CREATE/ALTER and replay; no new process setting requiring dynamic propagation was added. The existing force_jni_scanner and enable_file_scanner_v2 switches were checked for reachable paths.
  6. Compatibility: optional Thrift field 20 agrees across FE, BE, and the fixed fixture. The old-follower Fluss case is already in an existing thread; no other new rolling-format issue was substantiated.
  7. Parallel paths: legacy/plugin JDBC, native/JNI, Parquet/ORC, scalar/nested, TVF/table, and read/write counterparts were compared. Source-specific gaps are recorded separately in the inline findings.
  8. Conditions: JDBC wall-clock predicate guards and dropped-filter LIMIT handling, Paimon historical field-ID fallback, and static/hybrid NULL branches were traced; no additional incorrect guard survived.
  9. Test coverage: changed unit/integration and regression suites exercise common and selected negative cases, but miss the accepted official-driver, year-10000, mixed Iceberg binary/NULL, and nested MaxCompute cases.
  10. Expected results: changed .out fixtures were compared with ordered queries by static inspection; their execution and generation were not independently verified. Existing style comments cover fixed assertions and post-run drops.
  11. Observability: ordinary connector/BE errors retain identifying context, while the silent invalid timestamp packing and mislabeled Iceberg file are the issues reported here; no distinct metrics/logging gap was proven.
  12. Persistence/failover: marker migration journals before local apply and replay restores it after a crash; the residual mixed-version Fluss concern was deduplicated with the prior thread.
  13. Writes/atomicity: Iceberg can commit a file with rows from two logical partitions under one first-row partition value; nested MaxCompute zoned writes fail before completion. Other inspected write and cleanup paths showed no separate atomicity defect.
  14. FE/BE variables: the mapping markers and optional NULL-key Thrift field are passed to the relevant readers/writers; typed partition value and JNI timestamp transport were traced end to end.
  15. Performance/memory: projected/historical Paimon checks and connector conversions were inspected for repeated work and allocation; no distinct material regression was substantiated.
  16. Other issues: the final changed-file/accepted-anchor sweep found no unresolved candidate, while the final-round new issue leaves convergence incomplete as stated above.

Validation was static only as required: no build, test, or source edit was performed. The read-only BE header-hygiene gates passed; inspected test files and author/CI claims are not independent execution evidence.

Comment thread be/src/exec/sink/writer/iceberg/viceberg_table_writer.cpp

@924060929 924060929 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up FE review of 73dd4b8:

Preserving native UUID is the right direction. The previous UUID-typed input, recursive MAP conversion, MySQL double-minus, and NO_BACKSLASH_ESCAPES findings are addressed in this revision.

Please also revisit these three existing P2 threads; their triggering paths remain at this head:

I added one new inline comment on Trino JDBC UUID predicate literals. These concrete issues should be addressed before merging. This review is based on source and dependency inspection; no local build or runtime tests were run.

### What problem does this PR solve?

Related PR: apache#68786

Problem Summary: Render typed UUID literals for Trino and Presto JDBC predicates and canonical UUID values for Iceberg LIST partition metadata. Choose the legacy Paimon ORC reader from each file's historical schema so added nullable LTZ columns do not disable native physical metadata queries. Bind general UUID-bearing STRUCT inputs once before null testing and field conversion, preserving the single-evaluation invariant independently of later optimizer rewrites.

### Release note

Fix filtered Trino UUID reads, Iceberg UUID partition enumeration, and Paimon physical metadata queries after adding LTZ columns. Preserve single evaluation during nested UUID write conversion.

### Check List (For Author)

- Test: FE build; 396 related unit tests; JDBC predicate test rerun; FE Checkstyle; complete Iceberg UUID write and Trino UUID predicate regression suites, with generated output followed by an independent comparison run.
- Behavior changed: Yes, correct native UUID predicate/partition handling and historical ORC reader selection.
- Does this need documentation: No
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29203 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 37b315f775a3ce4fd80a539e37aab3d06a977f39, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17578	5700	5547	5547
q2	2085	296	253	253
q3	10269	1473	838	838
q4	4681	482	352	352
q5	7430	800	538	538
q6	191	187	152	152
q7	762	806	617	617
q8	9242	1362	1114	1114
q9	5764	4474	4502	4474
q10	6803	1300	1006	1006
q11	427	248	243	243
q12	629	415	287	287
q13	18052	3041	2379	2379
q14	272	267	256	256
q15	q16	736	726	660	660
q17	1168	736	579	579
q18	7095	6220	6171	6171
q19	1112	965	620	620
q20	384	337	225	225
q21	5562	2597	2664	2597
q22	416	356	295	295
Total cold run time: 100658 ms
Total hot run time: 29203 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	6673	6619	6764	6619
q2	752	621	595	595
q3	5551	5833	5244	5244
q4	2121	2167	1581	1581
q5	5739	5779	5705	5705
q6	252	195	154	154
q7	2340	2016	1748	1748
q8	3252	2812	2782	2782
q9	8113	8154	8069	8069
q10	3643	3617	3259	3259
q11	598	409	383	383
q12	680	698	513	513
q13	2732	3084	2392	2392
q14	284	281	268	268
q15	q16	657	734	616	616
q17	7793	7199	6920	6920
q18	13190	12319	13047	12319
q19	917	813	800	800
q20	2183	2182	1924	1924
q21	5907	4764	4953	4764
q22	494	470	401	401
Total cold run time: 73871 ms
Total hot run time: 67056 ms

### What problem does this PR solve?

Related PR: apache#68786

Problem Summary: Oracle TSLTZ scans did not initialize the driver's session time zone, and native Paimon scans discarded binary partition values needed by Fluss lake reads. Initialize each borrowed Oracle connection through its driver API and encode binary partition keys with the existing VARBINARY hex transport. Query TVFs also repeated schema prepares and fetched MySQL SQL mode even when no timestamp projection was needed. Reuse query schema within the statement, defer SQL-mode lookup until rewriting, and share its result between scan planning and EXPLAIN.

### Release note

Fix Oracle TSLTZ reads and Paimon binary partition values, and reduce remote metadata calls during JDBC query TVF planning.

### Check List (For Author)

- Test: Targeted FE/JDBC/Fluss/Paimon unit tests; real Paimon ORC/Parquet scan planning; Oracle integration tests with modern and legacy drivers across three JVM time zones; FE Checkstyle.
- Behavior changed: Yes, preserve typed partition values and initialize Oracle TSLTZ scans; avoid redundant JDBC planning IO.
- Does this need documentation: No
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@Gabriel39

Copy link
Copy Markdown
Contributor Author

Addressed the remaining four in-scope review findings in 3013084:

  • Initialize the Oracle driver session time zone on every borrowed scan connection, before reading TSLTZ values.
  • Preserve Paimon BINARY/VARBINARY partition keys as typed hex, including native ORC/Parquet ranges consumed by Fluss lake reads.
  • Read MySQL/OceanBase SQL mode only when a query TVF needs timestamp projection, and reuse it within the statement.
  • Reuse query TVF schema discovery across binding and scan-column handle construction.

Local validation: 321 targeted unit tests passed, plus two Oracle integration tests covering both current and legacy JDBC drivers under three JVM time zones. FE Checkstyle passed. Paimon tests create real ORC/Parquet files; the Fluss test covers matching lake/log partition values.

Resolved 58 review threads whose findings are fixed, superseded by native UUID mapping, or invalidated by the verified implementation. The 15 remaining threads concern explicitly excluded V1/compatibility behavior or unsupported VARBINARY functions and remain open. This does not claim those excluded findings are fixed.

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review recovery stopped: No final review submission was declared
Workflow run: https://github.com/apache/doris/actions/runs/38031271859

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review complete for head 30130849adb2423c897241f73b9745b521e4cc56 after two review rounds and a final sweep of all 350 changed paths. I found one new P2 issue, reported inline. Existing P1 comment 4217826038 still applies: the BE JDBC writer sends a numeric dialect to a JNI factory that selects handlers by name, skipping dialect-specific TIMESTAMPTZ write handling. I carried that comment ID without reposting it. The other examined P1 comments are fixed on this head. There was no additional user review focus.

Critical checkpoints:

  • Goal and scope: The PR moves external binary, UUID, and timestamp mappings through FE, BE, and JNI. The related adapters and regression coverage are broad but tied to that goal. The V1 ORC year-zero boundary remains incorrect.
  • Concurrency: Catalog migration runs under the catalog write lock; journal writes precede application and deferred cleanup runs after unlocking. No separate lock-order or race issue was substantiated.
  • Lifecycle: Master promotion, replay, catalog cache reset, and connector ownership were traced. No new lifecycle or static-initialization defect was substantiated.
  • Configuration: CREATE, ALTER, and promotion normalize the existing mapping markers to the new effective policy. No new dynamic setting was introduced; the existing scanner setting exposes the V1 ORC issue.
  • Compatibility: Mixed FE/BE UUID, Hudi timestamp, MaxCompute timestamp, and legacy FIXED paths are covered by existing inline threads. The existing JDBC writer P1 remains applicable; the old V1 UUID equality-delete P1 is fixed.
  • Parallel paths and conditions: V1/V2 file readers, native/JNI fallback, and partition read/write paths were traced. The new V1 lower-bound condition rejects a value V2 and Doris accept. Other investigated conditions were either fixed or already threaded.
  • Test coverage: New unit tests, regression suites, and expected outputs were inspected statically. The year-zero ORC fixture exercises V2 only; V1 boundary coverage is missing. No builds or tests were run, as required by this review invocation, so runtime results are unverified.
  • Test results: Changed suite labels and recorded outputs were checked for consistency; no distinct output mismatch was substantiated. The newly added JDBC writer test statically expects dialect names that the current BE path does not send.
  • Observability and errors: The changed readers and writers propagate errors with context; no independent logging or metrics gap was substantiated.
  • Persistence: The catalog marker migration journals before applying changes and reuses ALTER replay; promotion, failure, and idempotence test paths were inspected without a separate persistence finding.
  • Writes and crashes: Iceberg static and dynamic partition encoding, NULL/binary writer keys, and sink coercion were traced. The JDBC dialect mismatch above remains the write-side blocker; no other unthreaded atomicity or crash issue was substantiated.
  • FE/BE contract: No Thrift or protobuf definitions changed. Existing carriers and rolling-upgrade paths were reviewed; previously reported compatibility concerns were not duplicated.
  • Performance and other paths: Query-TVF planning overhead is already covered by existing threads. No further distinct performance or correctness issue survived validation and deduplication.

Existing P0/P1 findings confirmed for this head: #68786 (comment)

Comment thread be/src/format/orc/vorc_reader.h
@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29490 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 30130849adb2423c897241f73b9745b521e4cc56, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17572	5795	5678	5678
q2	2078	309	252	252
q3	10272	1518	875	875
q4	4679	490	354	354
q5	7449	822	542	542
q6	221	206	159	159
q7	825	847	627	627
q8	9430	1553	1193	1193
q9	5998	4668	4663	4663
q10	6909	1308	1008	1008
q11	454	259	249	249
q12	643	447	293	293
q13	18030	3197	2410	2410
q14	274	281	244	244
q15	q16	744	735	682	682
q17	1373	762	582	582
q18	7013	6279	6157	6157
q19	1107	1116	650	650
q20	418	358	236	236
q21	5406	2361	2521	2361
q22	390	315	275	275
Total cold run time: 101285 ms
Total hot run time: 29490 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	6390	6334	6363	6334
q2	739	571	535	535
q3	5292	5442	4970	4970
q4	2139	2230	1583	1583
q5	5328	5239	5161	5161
q6	287	209	150	150
q7	2173	1855	1686	1686
q8	3056	2692	2697	2692
q9	7770	7774	7752	7752
q10	3730	3644	3271	3271
q11	629	413	408	408
q12	710	745	514	514
q13	2871	3228	2403	2403
q14	290	290	272	272
q15	q16	692	728	635	635
q17	8105	7137	6970	6970
q18	13166	12386	13116	12386
q19	1073	880	888	880
q20	2227	2201	1974	1974
q21	6226	4886	4983	4886
q22	537	473	419	419
Total cold run time: 73430 ms
Total hot run time: 65881 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 152479 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 30130849adb2423c897241f73b9745b521e4cc56, data reload: false

query5	4213	571	449	449
query6	414	210	189	189
query7	4790	450	223	223
query8	315	185	172	172
query9	8356	4102	4071	4071
query10	468	323	264	264
query11	3715	3458	3220	3220
query12	131	91	88	88
query13	1159	441	325	325
query14	5209	4825	4504	4504
query14_1	4242	4254	4263	4254
query15	197	209	177	177
query16	574	433	421	421
query17	619	669	546	546
query18	420	424	318	318
query19	188	166	139	139
query20	84	84	80	80
query21	146	143	115	115
query22	13078	13149	12889	12889
query23	13248	12426	12161	12161
query23_1	12289	12243	12281	12243
query24	7581	825	464	464
query24_1	479	483	477	477
query25	601	420	357	357
query26	1651	263	147	147
query27	2804	447	277	277
query28	4657	1948	1939	1939
query29	3583	630	458	458
query30	298	217	190	190
query31	932	766	646	646
query32	153	95	90	90
query33	527	311	253	253
query34	1102	862	500	500
query35	746	788	676	676
query36	796	829	737	737
query37	214	105	92	92
query38	1803	1751	1701	1701
query39	735	714	684	684
query39_1	687	643	663	643
query40	299	121	134	121
query41	64	60	60	60
query42	80	83	85	83
query43	369	386	317	317
query44	1287	702	701	701
query45	186	177	158	158
query46	853	960	579	579
query47	3042	2867	2842	2842
query48	296	304	211	211
query49	580	400	294	294
query50	687	272	204	204
query51	10413	10411	10515	10411
query52	82	78	75	75
query53	189	202	152	152
query54	233	192	182	182
query55	74	68	67	67
query56	250	211	216	211
query57	1607	1600	1547	1547
query58	272	268	251	251
query59	2372	2373	2104	2104
query60	287	226	220	220
query61	148	145	136	136
query62	387	337	286	286
query63	202	159	166	159
query64	2752	930	804	804
query65	3454	3381	3361	3361
query66	1961	415	301	301
query67	20282	20341	20065	20065
query68	3017	1010	605	605
query69	378	291	255	255
query70	915	844	847	844
query71	293	223	204	204
query72	2785	2537	2257	2257
query73	508	528	310	310
query74	4596	4473	4298	4298
query75	2377	2286	1954	1954
query76	2342	1028	620	620
query77	367	394	295	295
query78	9098	8979	8502	8502
query79	1019	861	512	512
query80	531	414	360	360
query81	523	320	270	270
query82	536	138	110	110
query83	301	202	174	174
query84	278	125	95	95
query85	830	418	366	366
query86	361	240	229	229
query87	1998	1978	1902	1902
query88	3626	2715	2689	2689
query89	397	294	250	250
query90	1761	189	183	183
query91	164	154	126	126
query92	97	86	88	86
query93	1057	990	565	565
query94	460	332	285	285
query95	594	339	335	335
query96	650	524	230	230
query97	2444	2445	2321	2321
query98	155	149	145	145
query99	786	761	641	641
Total cold run time: 227175 ms
Total hot run time: 152479 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.52 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 30130849adb2423c897241f73b9745b521e4cc56, data reload: false

query1	0.01	0.01	0.00
query2	0.10	0.05	0.05
query3	0.27	0.11	0.12
query4	1.60	0.12	0.13
query5	0.28	0.25	0.24
query6	1.18	0.69	0.68
query7	0.04	0.01	0.01
query8	0.05	0.04	0.04
query9	0.41	0.35	0.34
query10	0.57	0.61	0.58
query11	0.21	0.14	0.14
query12	0.18	0.15	0.15
query13	0.48	0.49	0.51
query14	0.97	0.98	0.97
query15	0.64	0.61	0.62
query16	0.36	0.32	0.35
query17	1.13	1.10	1.13
query18	0.23	0.20	0.21
query19	2.09	2.01	1.95
query20	0.02	0.01	0.01
query21	15.45	0.27	0.15
query22	4.50	0.05	0.05
query23	16.24	0.32	0.12
query24	3.13	0.59	0.52
query25	0.10	0.04	0.04
query26	1.21	0.27	0.15
query27	0.06	0.04	0.04
query28	2.00	0.60	0.43
query29	12.51	4.14	3.28
query30	0.28	0.13	0.14
query31	2.78	0.59	0.34
query32	3.22	0.62	0.52
query33	3.29	3.13	3.25
query34	15.45	3.99	3.40
query35	3.32	3.35	3.32
query36	0.55	0.47	0.44
query37	0.09	0.06	0.07
query38	0.05	0.04	0.03
query39	0.03	0.03	0.03
query40	0.18	0.15	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 95.43 s
Total hot run time: 24.52 s

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of PR #68786 at head 3013084 (350 changed paths). Four new inline findings: one P1 and three P2. All 78 existing inline comments were checked for duplicates. Ten earlier P1 comments were independently rechecked and no longer apply at this head; there are no carried P0/P1 blocking IDs. The user supplied no additional review focus.

Review completion: All current candidate findings have been resolved and the final changed-file sweep found no uncovered path. Convergence remains incomplete: new distinct findings appeared in round 3, and the required three-round cap prevents a further convergence pass. This is a static review; no build, test, or runtime validation was performed.

Critical checkpoints

  • Goal and proof: The PR aims to preserve native binary, UUID, and timestamp semantics across external connectors and writes. The four inline failures show that the Iceberg BINARY path does not yet meet that goal. The added tests inspected do not prove these V1 delete, set operation, window, or nested-key cases.
  • Scope and clarity: The migration spans 350 paths across BE, FE, connector Java, and regression suites. The changed type mapping is clear, but its effects on unchanged hash consumers are not fully covered.
  • Concurrency: Catalog promotion, replay, scan routing, and writer-key paths were traced. No additional lock-order or shared-state race was substantiated in the inspected paths.
  • Lifecycle: Migration journals markers before master readiness; scan and writer close paths were checked. No separate lifecycle or cross-translation-unit static-initialization failure was established.
  • Configuration dynamics: Legacy mapping flags are normalized during promotion and ALTER; current code makes the new logical mapping effective without relying on a dynamic flag refresh. No distinct unreported configuration issue was established.
  • Compatibility: Rolling-version, stored catalog, view, and FE/BE type contracts were checked against existing threads. Previously raised P1s are fixed on this head; the new Iceberg BINARY failures are separate compatibility regressions.
  • Generated wire definitions: No gensrc/thrift or gensrc/proto file changed, so field-ID and enum compatibility checks do not apply.
  • Parallel paths: V1/V2 file readers, Iceberg delete paths, JDBC scanner/writer routes, Paimon native/JNI routes, and BE set/hash dispatch were compared. The four inline issues are the distinct missing paths found.
  • Conditional checks: FE guards reject scalar VARBINARY grouping and joins, but omit set operations and window partitioning and do not recurse into array element types. The related BE hooks still throw; see inline findings.
  • Coverage: Existing regression and unit changes were inspected, including expected-output files. Targeted positive and negative coverage is missing for the four reported execution paths.
  • Test results: Expected-output changes were reviewed statically only. Their runtime correctness cannot be certified because the review instructions prohibited builds and tests.
  • Observability: The failing BE paths return explicit unsupported-operation errors. No additional logging or metric defect was substantiated.
  • Persistence: Catalog migration journal ordering, replay, and master promotion were traced; no separate unjournaled-state defect was established.
  • Writes and atomicity: Typed Iceberg writer keys, nullness separation, static partition metadata, and commit paths were checked. Previously reported write issues are fixed or already covered by existing comments; no new atomicity defect was established.
  • FE/BE variables: Mapping options, schema context, UUID/binary types, timestamp handling, and partition nullness were followed across FE/BE carriers. No separate missing carrier was established.
  • Performance: Scan selection, JNI projection, partition routing, and hashing paths were examined. No distinct CPU, memory, or asymptotic regression was substantiated beyond the functional failures above.
  • Other issues: Remaining candidate points were either duplicates of existing inline comments or dismissed with concrete downstream evidence. No unresolved candidate remains within the completed three rounds.

return enableMappingVarbinary
? ConnectorType.of("VARBINARY") : ConnectorType.of("STRING");
// Binary payloads need not be valid UTF-8.
return ConnectorType.of("VARBINARY");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Keep V1 equality deletes on BINARY fields readable. With enable_file_scanner_v2=false, the new BINARY-to-VARBINARY mapping reaches both V1 delete implementations. A one-field equality delete calls create_set(TYPE_VARBINARY) and throws NOT_IMPLEMENTED_ERROR; a composite delete containing BINARY calls ColumnVarbinary::update_hashes_with_value through ColumnNullable, which also throws because the hash hook is unsupported. Before this change BINARY mapped to STRING by default, so these scans worked. The existing UUID thread concerns a different type that now has native TYPE_UUID set support. Add V1 BINARY equality-delete coverage for single and composite keys and provide byte-aware delete handling.

return enableMappingVarbinary
? ConnectorType.of("VARBINARY") : ConnectorType.of("STRING");
// Binary payloads need not be valid UTF-8.
return ConnectorType.of("VARBINARY");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep INTERSECT and EXCEPT on BINARY columns executable. Two Iceberg BINARY inputs now retain VARBINARY through set-operation binding and produce IntersectNode or ExceptNode, but SetSharedState::hash_table_init calls get_hash_key_type, which throws NOT_IMPLEMENTED_ERROR for VARBINARY. For example, SELECT payload FROM t1 INTERSECT SELECT payload FROM t2 now fails at BE open, whereas the former default STRING mapping could build the set. The FE VARBINARY checks cover scalar grouping and joins, but not these set operations. Add byte-aware set hashing or reject these operands during analysis, with INTERSECT and EXCEPT coverage.

return enableMappingVarbinary
? ConnectorType.of("VARBINARY") : ConnectorType.of("STRING");
// Binary payloads need not be valid UTF-8.
return ConnectorType.of("VARBINARY");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep windows partitioned by BINARY executable. A query such as row_number() OVER (PARTITION BY payload ORDER BY id) on an Iceberg BINARY column still passes FE analysis, but the window requires a hash exchange on the newly mapped VARBINARY key. The BE partitioner calls ColumnVarbinary's unsupported CRC hash hook and fails; with a filtered row-number query, the default PartitionTopN rewrite also reaches get_hash_key_type(TYPE_VARBINARY) and fails. The former default STRING mapping supported both paths. Add byte-aware window partition hashing or reject this key at analysis, and cover ordinary and filtered window plans.

return enableMappingVarbinary
? ConnectorType.of("VARBINARY") : ConnectorType.of("STRING");
// Binary payloads need not be valid UTF-8.
return ConnectorType.of("VARBINARY");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Handle VARBINARY inside array hash keys. Iceberg LIST now becomes ARRAY, but GROUP BY and DISTINCT only reject a top-level VARBINARY key. A distributed SELECT payload, COUNT(*) FROM t GROUP BY payload therefore requests a hash exchange on the array; for any nonempty payload, ColumnArray delegates CRC hashing to ColumnVarbinary, whose hook throws NOT_IMPLEMENTED_ERROR. The former ARRAY key worked. The same top-level-only guard admits equality joins on arrays. Provide nested byte hashing or recursively reject these keys during FE analysis, and cover GROUP BY, DISTINCT, and joins on nonempty arrays.

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review conclusion

Static review of PR #68786 at head 30130849adb2423c897241f73b9745b521e4cc56. The patch moves external binary, UUID, and instant values to their native types across FE schemas, native BE readers/writers, and JNI connectors. I found the inline issues below in current reads and in both rolling-upgrade directions. The active V1 BINARY equality-delete P1 is already reported in comment 4237058215; I confirmed it still applies and carried its ID without reposting it. Older UUID equality-delete and year-zero scanner-test P1 comments are fixed at this head. There was no additional user review focus.

Convergence status: incomplete. The capped third full-review round found the converted-only Parquet timestamp issue. All known candidates from the three rounds were checked and resolved into an inline finding, an existing thread, or a concrete dismissal, but the required no-new-finding convergence was not reached before the three-round limit.

Critical checkpoints

  • Goal and proof: Native types preserve intended binary bytes, UUID semantics, and timestamp instants on several new paths, but the reported file-reader, JDBC, native connector, and mixed-version failures prevent the change from meeting that goal for all supported data. New unit and regression cases cover many current-version types; they do not prove the mixed-version and historical-file cases reported here.
  • Scope and focus: Reviewed the authoritative 350 changed paths across BE, FE connectors/core/SPI, Java extensions, and regression output. The cross-module scope follows the type-contract change; no separate user focus was supplied.
  • Concurrency and locking: Catalog marker migration journals and applies under the catalog global write lock, with deferred cache cleanup after release. No new lock-order, race, or heavy work inside a lock was substantiated beyond already threaded old-follower behavior.
  • Lifecycle: Checked master promotion/replay, catalog cache reset, Paimon fallback split and iterator ownership, JDBC connection initialization, and BE writer open/close. The Oracle-mode OceanBase initialization and old-BE ORC writer assertion are the actionable lifecycle failures; no separate ownership or static-initialization defect was found.
  • Configuration: Legacy mapping flags become effectively enabled and are migrated through the catalog path. This exposes native FE slots while old BEs still serve scans/writes; no BE capability gate is present for the reported paths. Existing threads cover old-follower, stored-view, and MTMV consequences of the metadata migration.
  • Compatibility: Both upgrade directions and historical file annotations were traced. Existing threads cover several old-FE/new-BE cases; the new inline findings cover distinct FE-first/old-BE consumers and current-BE Hudi/Parquet representation gaps. No gensrc/thrift or gensrc/proto file changed, so there is no new field-ID contract to audit.
  • Parallel paths: Checked V1/V2 Parquet and ORC, Hudi COW/MOR, native Trino and JDBC scans, MaxCompute read/write, Iceberg partitioned and unpartitioned sinks, and Paimon/Fluss/Hive paths. The reported issues are distinct in their consumer or physical carrier; remaining plausible parallel cases were covered by existing threads or had matching dispatch.
  • Conditions and error handling: The old ORC UUID path treats a newly valid FE type as an invariant violation and can abort a BE. Other unsupported conversions/handler cases surface errors rather than being silently ignored; the timestamp read/bind cases can silently shift instants. No further ignored Status or catch boundary issue was substantiated.
  • Tests and results: Inspected changed unit/regression cases and .out entries statically, including negative and ordered-result cases. Mixed-version scans/inserts, Hudi COW Parquet UUID, and converted-only Parquet timestamp files need coverage. The review prompt prohibits builds and tests, so no runtime pass claim is made.
  • Observability: Existing conversion and JNI errors expose the failure stage; no independent missing log or metric issue was substantiated. Silent timestamp shifts need the proposed non-UTC tests.
  • Transactions and persistence: Catalog migration journals before apply and follows replay on promotion. Existing threads report remaining old-follower and persisted-object consistency effects. No additional journal/replay mismatch was established.
  • Writes and atomicity: Iceberg and MaxCompute mixed-version writes can fail before file/commit completion; PostgreSQL JDBC can persist the wrong instant on an old BE with a non-UTC JVM. No additional transaction atomicity, leak, or commit-metadata defect was substantiated on current BEs.
  • FE–BE transport: Existing slot and JNI columns_types transport conveys the new types, but the old per-connector handlers and writers lack matching support. No new wire variable or field was added; capability/version handling is the missing contract.
  • Memory, nullability, and data correctness: Reviewed BE nullable/const handling, UUID SerDe and Arrow ownership, partition NULL keys, binary hash/JSON consumers, and type conversion. No independent allocator, lifetime, or nullable-shape issue survived review. The existing V1 BINARY equality-delete P1 and the new type/instant issues are the data-correctness blockers.
  • Performance and other risks: The JDBC SQL-mode round-trip concern is already in an existing thread. No further distinct CPU, memory, or scalability regression was substantiated in this static pass.

Existing P0/P1 findings confirmed for this head: #68786 (comment)

return enableMappingVarbinary
? ConnectorType.of("VARBINARY", 16, 0) : ConnectorType.of("STRING");
// Preserve logical UUID semantics independently of the binary mapping option.
return ConnectorType.of("UUID");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep UUID reads working while BEs are upgraded. A new FE now plans Iceberg UUID as native UUID, but a pre-upgrade BE V1 Parquet reader maps a UUID-annotated file to STRING or VARBINARY and has no conversion from either carrier to a UUID slot. Thus an FE-first upgrade breaks ordinary UUID scans until every BE is replaced. Preserve the old carrier while older BEs can receive scans, or gate native UUID plans on BE capability; cover this upgrade direction. The existing thread covers the reverse old-FE/new-BE direction.

// ALTER SESSION alone leaves the driver's TSLTZ zone unset. Initialize every borrowed connection.
try {
Connection physical = conn.unwrap(Connection.class);
Class<?> oracleConnection = Class.forName("oracle.jdbc.OracleConnection", true,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Preserve Oracle-mode OceanBase scans. OCEANBASE_ORACLE selects this handler, but those catalogs use com.oceanbase.jdbc.Driver. This unconditional lookup and unwrap require oracle.jdbc.OracleConnection from the OceanBase driver classloader before any SQL is prepared, so ordinary scans fail with the session-time-zone initialization error, even when no TIMESTAMPTZ column is read. Apply this Oracle-specific call only to Oracle JDBC connections and use an OceanBase-compatible setup for its mode; cover an Oracle-mode OceanBase scan.

// DateTime('timezone') — DateTime with timezone parameter, second precision
if (chType.startsWith("DateTime(")) {
return ConnectorType.of("DATETIMEV2", 0, -1);
return ConnectorType.of("TIMESTAMPTZ", 0, -1);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep JDBC scans readable while BEs are upgraded. This new FE mapping sends TIMESTAMPTZ for ClickHouse DateTime, but an older BE JDBC scanner parses that slot and its ClickHouseTypeHandler has no TIMESTAMPTZ branch, so the first row throws Unsupported column type. The new PostgreSQL/Trino UUID mappings have the same old-handler gap. Preserve the previous carriers until all BEs have the new handlers, or gate these plans on BE capability; cover a new-FE/old-BE scan. This is separate from the Iceberg file-reader upgrade issue.

case TIMESTAMP:
if (enableMappingTimestampTz
&& ((Types.TimestampType) primitive).shouldAdjustToUTC()) {
if (((Types.TimestampType) primitive).shouldAdjustToUTC()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve partitioned Iceberg INSERTs during an FE-first upgrade. A new FE now sends TIMESTAMPTZ (and native UUID) write slots, but an older BE has no cases for these types in Iceberg partition transforms or _get_iceberg_partition_value. An identity-partitioned write reaches Unsupported type for partition; bucket/time transforms can fail earlier. Gate typed write plans on BE capability or keep compatible carriers until all BEs are upgraded, and cover a mixed-version partitioned INSERT. The existing UUID scan issue is a separate reader path.

switch (nested_field->field_type()->type_id()) {
case iceberg::TypeID::UUID:
// Native UUID serde already writes network-order bytes; retain its ORC annotation.
if (primitive_type == TYPE_UUID) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Gate native UUID Iceberg ORC writes until every BE supports them. An upgraded FE now sends TYPE_UUID for an Iceberg UUID column, including on an unpartitioned ORC table. An older BE reaches use_iceberg_binary_type in VOrcTransformer::_build_orc_type, whose DORIS_CHECK accepts only string, varbinary, or binary, and aborts the BE while opening the writer. This new TYPE_UUID branch exists only on upgraded BEs. Retain the old write carrier or gate this plan on BE capability; cover an FE-first unpartitioned ORC INSERT.

if (logicalType instanceof LogicalTypes.TimestampMillis) {
return ConnectorType.of("DATETIMEV2", 3, 0);
// Avro timestamp logical types are instants, not local wall-clock timestamps.
return ConnectorType.of("TIMESTAMPTZ", 3, 0);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Gate Hudi instant slots until older MOR scanners are gone. This new TIMESTAMPTZ mapping calls HadoopHudiColumnValue.getTimeStampTz on an old BE; that method casts every value to java.sql.Timestamp. Hudi also supplies LongWritable and TimestampWritableV2 timestamp carriers, which throw ClassCastException there on ordinary MOR log scans. The added branches handle them only on new BEs. Keep the previous carrier during rollout or gate this mapping; cover both writable carriers in a mixed-version scan.

case "DATETIME":
case "DATETIMEV2":
return TypeInfoFactory.DATETIME;
case "TIMESTAMPTZ":

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep MaxCompute TIMESTAMP INSERTs working during FE-first upgrades. A new FE now sends TIMESTAMPTZ for this sink column, but the old MaxComputeJniWriter handles TIMESTAMP in its DATETIME branch and calls VectorColumn.getDateTime. That decodes the TIMESTAMPTZ V2 carrier as DateTimeV1, producing invalid fields and failing before the Arrow write. The new getTimeStampTz branch exists only on upgraded BEs. Gate this sink slot or keep its old carrier until writers are upgraded.

public boolean isEnableMappingTimestampTz() {
return enableMappingTimestampTz;
// Instant types cannot be downgraded to session-local wall clocks.
return true;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve JDBC TIMESTAMPTZ writes on older BEs. For a PostgreSQL catalog that previously disabled zoned mapping, this now forces a TIMESTAMPTZ sink slot. An old BE writer binds its UTC JNI fields with Timestamp.valueOf, which interprets them in the BE JVM timezone; with a +08 JVM, 04:00 UTC is sent as the previous day 20:00 UTC. The new UTC/OffsetDateTime bind is only on upgraded BEs. Gate this slot on writer capability or retain the compatible carrier until rollout finishes; cover a non-UTC JVM write.

return ConnectorType.of("STRING");
// Avro stores logical UUIDs as strings, but the connector must retain UUID semantics.
return logicalType instanceof LogicalTypes.Uuid
? ConnectorType.of("UUID") : ConnectorType.of("STRING");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve Hudi COW UUID scans for Parquet Avro string files. Avro logical uuid is carried as STRING, and the default Parquet Avro writer stores it as a BINARY STRING leaf; COW base files use Doris's native Parquet reader. This new UUID slot then asks the V1 reader to convert file STRING to UUID, but ColumnTypeConverter has no such conversion and returns Unsupported type change even on an upgraded BE. The new UUID decoder only covers UUID-annotated 16-byte fixed fields. Keep the compatible string mapping for these files or add canonical text-to-UUID conversion in native readers, and test a default-written COW UUID file.

return ConnectorType.of("TIMESTAMPTZ", 3, 0);
}
if (logicalType instanceof LogicalTypes.TimestampMicros) {
return ConnectorType.of("TIMESTAMPTZ", 6, 0);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Read historical Hudi COW Parquet timestamps with the new instant slot. Hudi files written with Parquet Java 1.10.1 can have INT64 TIMESTAMP_MILLIS/MICROS in converted_type without the newer LogicalType field. COW routes those files to the native reader, which still infers DATETIMEV2; V1 has no DATETIMEV2-to-TIMESTAMPTZ conversion for this new FE slot and fails an ordinary scan with Unsupported type change, even on an upgraded BE. Preserve a UTC-aware conversion for legacy footers in V1/V2 or keep a compatible slot, and add a converted-only file fixture.

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 27.71% (179/646) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 68.59% (190/277) 🎉
Increment coverage report
Complete coverage report

@Gabriel39
Gabriel39 merged commit a01cc89 into apache:master Oct 10, 2026
38 of 41 checks passed
morningman added a commit to morningman/doris that referenced this pull request Oct 10, 2026
…Z binding only

### What problem does this PR solve?

Issue Number: None

Related PR: apache#65868, apache#68786

Problem Summary: apache#68786 binds every Paimon TIMESTAMP WITH LOCAL TIME ZONE
column as TIMESTAMPTZ; enable.mapping.timestamp_tz no longer selects a
DATETIMEV2 binding. The cast value of a static LTZ partition is therefore
always the instant in UTC with its offset, and the session-zone path of
PaimonWriteBinding, with its check for a session time inside a DST gap,
can no longer run. That check existed because the BE turned a DATETIMEV2
row value into an instant with cctz while the FE used java.time; a
TIMESTAMPTZ row value and the static value now come from the same FE cast.
Remove the path and the session zone it needed, and keep the rejection of
an instant whose local time the FE JVM zone repeats.

PaimonConnectorTransactionTest, added by apache#68858 after this change was
written, builds the write binding through the new create signature.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - PaimonWriteBindingTest: the LTZ zone move, the overlap rejection and
      the round trip through Paimon's parser, all for TIMESTAMPTZ values.
    - The Paimon connector (701) and fe-connector-spi (147) tests, and the
      fe-core binding, BindSink and insert command tests (52) pass.
- Behavior changed: No (the removed path had become unreachable)
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants