4.0.5 release notes

4.0.6 Release Notes

Apache Doris 4.0.6 is a maintenance release in the 4.0 series. It focuses on stability and operational experience. All 4.0.x users are advised to upgrade, and users running the compute-storage decoupled (cloud) mode benefit the most.

Highlights of this release include:

• Compute-storage decoupling enhancements: support read-write separation for Compaction, optimize Tablet scheduling and FE-side memory usage, and reduce RPC pressure on the Meta Service.
• Operational observability: add Compaction task tracking (System Table and HTTP API), a dynamic enable_recycler switch, online adjustment of the memtable flush thread pool, and several brpc / bvar metrics.
• Data lineage: introduce a Lineage SPI collection framework.
• Many correctness and stability fixes: cover incorrect query results, load data loss (such as Broker Load loading only the first file, and PostgreSQL DML being silently dropped), crashes, hangs, and resource leaks.

Before upgrading, read the Behavior Changes section below, which contains several adjustments to default values and privileges.

Behavior Changes

• In compute-storage decoupled (cloud) mode, enable_strict_consistency_dml is now disabled by default (branch-4.0: [opt](load) disable enable_strict_consistency_dml by default in cloud mode #61891). If your workload relies on strict-consistency DML, explicitly enable this configuration after upgrading.
• The default trigger threshold for Time Series Compaction is changed from 2000 to 1000 ([chore](compaction) change time series threshold from 2000 to 1000 #61979), so Compaction triggers more aggressively.
• The default value of segments_key_bounds_truncation_threshold is changed to 36 ([enhancement](config) Change segments_key_bounds_truncation_threshold default to 36 #61984).
• Cluster Snapshot commands now require root privileges ([Feat](Snapshot) require root privilege for cluster snapshot commands #60239).
• JSONB and Variant columns are no longer allowed as Distribution Columns ([fix](fe) Reject JSONB and variant distribution columns #63211). Such previously-incorrect usage is now rejected at table-creation time.

New Features & Improvements

Functions & Types

• Add the mmhash3_u64_v2 hash function ([Enhancement](mmhash) Support mmhash3_u64_v2 #61846).
• Add the json_object_flatten scalar function for flattening nested JSON ([feature](jsonb) add json_object_flatten scalar function #62825).
• Add a Lambda expression (functor) version of array_sort that supports custom sorting logic ([enhance](function) add a lambda functor version for array_sort #57828).
• The max / min aggregate functions now support the Array type, and max_by / min_by support some complex types ([enhance](agg) Support max/min agg functions for type_array #58490, [enhance](agg) Support max_by/min_by agg functions for some complex type #58736).

Query Optimization

• Limit the cost of not-null inference to reduce optimizer planning overhead for complex queries ([opt](fe) Bound not-null inference cost #63318).
• Scale the num_nulls column statistic proportionally after partition pruning to improve row-count estimation accuracy ([opt](nereids) scale num_nulls in col stats when partition pruned #62265).

Storage & Compaction

• Add Compaction task tracking: query current Compaction tasks through a System Table and an HTTP API ([Feature](compaction) add CompactionTaskTracker with system table and HTTP API #61696).
• Support adjusting the memtable flush thread pool size online, without restarting BE ([enhance](memtable) support dynamic modification of flush thread pool size #60423).
• Support adaptive Write Buffer sizing (branch-4.0: [enhance](memtable) support adaptive write buffer size #61810).
• Release unused memory earlier to reduce memory usage during load and write (branch-4.0：[improvement](storage) Release unused memory earlier, rather than leaving it to the memory pool for guaranteed release #62185).

Compute-Storage Decoupling (Cloud Mode)

• Support read-write separation for Compaction ([feature](cloud) Support compaction read-write separation for cloud mode  #60310).
• Schedule recently-active Tablets first to improve cache hit rate and scheduling efficiency ([opt](cloud) Prioritize scheduling the most recently active tablets in cloud #59539, [opt](cloud) Exposes cloud balance metrics  #57200, [fix](cloud) Fix test_balance_metrics docker case (#7885) #61562).
• Reduce get_tablet_stats RPC calls to the Meta Service to lower metadata-service pressure ([improve](cloud) cloud reduce get_tablet_stats rpc to meta_service #60543).
• Optimize the memory usage of CloudTabletStatMgr / CloudTabletRebalancer ([fix](fe) modify CloudTabletStatMgr to reduce memory #59776, [fix](cloud) modify CloudTabletRebalancer and CloudTabletStatMgr to reduce memory #61318).
• Add the enable_recycler configuration to dynamically skip the Recycler ([improvement](cloud) Add enable_recycler config to skip recycler dynamically #63286).
• Support hot-reloading of File Cache microbench configuration ([opt](filecache) support config hot reload of file cache microbench in running state #58922).

Lakehouse

• Iceberg supports IAM Role authentication for REST and S3 Tables ([feat](Iceberg)Rest & S3Table Support Iam-role #60498).
• Paimon supports Jindo OSS and Token propagation ([fix](paimon) add jindo oss support and token propagation #62106).
• Support disabling View operations for the Iceberg REST Catalog ([fix](iceberg) Allow disabling REST catalog view operations #63319).

Data Lineage

• Introduce a Lineage SPI framework to support data lineage collection ([feature](lineage) Support lineage SPI framework for data lineage collection #61004).

Observability & Operations

• Add the --drop_backends parameter to start_fe.sh ([fix](fe) add --drop_backends param to start_fe.sh #63306).
• Add a transaction write-amplification brpc metric for Sub Txn Load ([feat](txn) Add txn write amplification brpc metrics for sub txn load #63545).
• Add bvar metrics for the Recycler Operation Log ([Enhancement](Recycler) Add bvar metrics for operation logs #60520).
• Statistics collection now automatically skips overly-long string columns to reduce collection overhead ([feature](statistics) Skip collecting stats for long string columns #62686).
• Remove the class histogram trace from the JDK17 startup parameters to avoid printing large amounts of logs during Full GC ([opt](fe)remove classhisto*=trace in JAVA_OPTS_FOR_JDK_17 to prevent printing class histogram in full gc #62422).

Security & Authentication

• Hide Token and authentication information in BE Info during Stream Load ([enhance](auth)Hidden the token info during StreamLoad in BE info #60656, [enhance](auth)Hidden the auth info during StreamLoad in BE info #59743).
• Improve the robustness and diagnostics of LDAP authentication ( [fix](ldap) Improve LDAP authentication resiliency and diagnostics #61673).
• Upgrade dependencies with known security vulnerabilities ([fix](fe-dep) upgrade vulnerable dependencies #62274).

Important Bug Fixes

Query Result Correctness

• Fix INTERSECT / EXCEPT losing NULL rows during predicate pushdown ([fix](fe) Fix PullUpPredicates losing NULL rows in INTERSECT/EXCEPT #62299).
• Fix count(null) being incorrectly treated as count(*) ([fix](fe) Fix isCountStar incorrectly treating count(null) as count(*) #62548).
• Fix a NoSuchElementException when count contains a MATCH_ALL expression ([fix](fe) Fix NoSuchElementException when count with MATCH_ALL expression #62172).
• Fix incorrect results caused by losing a DateTimeV2 narrowing Cast during IN predicate simplification ([fix](fe) Preserve narrowing datetimev2 casts in simplify in predicate #63343).
• Fix loss of the IN_LIST Runtime Filter predicate in Key Range scenarios ([Bug](scan) Preserve IN_LIST runtime filter predicates when key range… #62115).
• Fix materialized view rewriting merging partitions that the query does not need, which could cause incorrect results or performance issues ([fix](mv) Avoid unioning query-unused MV partitions #63081).
• Fix the no_use_cbo_rule Hint being silently ignored ([fix](fe) Fix no_use_cbo_rule hint being silently ignored in EliminateLogicalSelectHint #62358).
• Fix FragmentMgr incorrectly canceling queries on a compute-storage decoupled Virtual Cluster ([fix](doris catalog) FragmentMgr should not cancel virtual doris cluster query #62135).

Views

• Fix ALTER VIEW definitions not being synchronized to Follower FEs when a COMMENT is specified ([fix](view) ALTER VIEW definition not synced to follower FE when COMMENT is specified #61670).
• Fix incorrect query results caused by a View definition losing Variant subfields ([fix](analyze) Preserve variant subfields in view definitions to fix select view result wrong when view select has variant field #62907).
• Fix View columns losing colUniqueId under Lazy Materialization ([fix](fe) Fix view columns losing colUniqueId in lazy materialization #62533).
• Fix illegal Alias rewriting in View definitions ([fix](view) Prevent invalid alias rewrite in view definitions #63353).

Functions & Types

• Fix several incorrect results related to TIMESTAMPTZ and Daylight Saving Time (DST): LEAD/LAG not preserving the type, spring-forward gap handling, DST fold branch selection, switching time-difference calculation to UTC semantics, and TopN Runtime Predicate support ([fix](functions)Preserve TIMESTAMPTZ in LEAD/LAG by adding TimeStampTzType signature matching #62779, [fix](nereids) Fix DST spring-forward gap handling in timestamptz literal #62945, [fix](timestamptz) Preserve correct DST fold branch to go cross the transition point #63034, [Fix](ttz) Fix TIMESTAMPTZ elapsed-time semantics to use UTC #63161, [fix](be) Support TIMESTAMPTZ in TopN runtime predicate #63220).
• Fix incorrect results for the allow_zero_date function ([fix](date) fix allow_zero_date function incorrect result issue #61900).
• Fix incorrect results when casting a scientific-notation string to Decimal ([fix](decimal) Fix incorrect decimal cast results for scientific-notation strings #63119).
• Fix from_olap_string throwing an exception when it fails to parse a datetime; it now returns NULL instead ([fix](parse) No longer throws exceptions when parse datetime failed in from_olap_string #63035).
• Fix incorrect integer inference and type resolution for user variables ([fix](user_var)fix integer typing and prefer Variable.realExpression for argument/type resolution #62524).

Variant Type

• Fix Compaction failure when a Variant column has uid=0 in a keyless table ([branch-4.0](variant) Fix compaction failure on no-key table with variant column uid=0 #62656).
• Fix Variant subcolumns being lost after a partial-column update in the Row Store ([fix](variant) preserve subcolumns after row-store partial update #62067).
• Fix a normalization issue when reading older-version single-segment dot-key subcolumns ([fix](variant) normalize legacy single-part dot-key subcolumn paths on read #62409).
• Keep the first occurrence when a Variant JSON Path is duplicated (cherry-pick branch-4.1 : [Improve](Variant) Keep first duplicate Variant JSON path #63697).
• Optimize VariantStatsCaculator construction by skipping the full footer scan to improve performance ([fix](variant) Skip full footer scan when constructing VariantStatsCaculator #62819).

Load & Streaming

Data Loss / Integrity

• Fix Broker Load loading only the first file when multiple file paths are specified ([fix](load) fix broker load silently loaded only the first file when parsing multiple files path #62969).
• Fix PostgreSQL DML being silently dropped after a job restart ([fix](streamingjob) fix postgres DML silently dropped on task restart #61481).
• Fix loss of S3 Offset and job statistics after an FE Checkpoint restart ([fix](streaming-job) fix S3 offset and job statistics lost after FE checkpoint restart #62449).
• Fix incorrect Java type restoration of Split boundaries when reading FE-persisted CDC Offsets ([fix](streaming-job) restore split-bound Java types when reading FE-persisted CDC offset #63219).

Hangs / Leaks

• Fix a VNodeChannel close_wait hang ([fix](load)fix VNodeChannel close_wait hang #58024).
• Fix a PostgreSQL Replication Slot leak caused by canceling a streaming job during pause/resume ([fix](streaming-job) Fix PG replication slot leak when streaming task is cancelled during pause/resume  #62010).
• Fix a memory leak caused by an InsertLoadJob being stuck in PENDING (branch-4.0: [fix](insert) fix InsertLoadJob memory leak caused by jobs permanently stuck in PENDING state #62890).

Restart & Metadata

• Fix streaming job properties not being parsed after an FE restart ([fix](streaming-job) fix streaming job properties not parsed after FE restart #62298).
• Fix a StreamingInsertJob NPE during EditLog Replay ([fix](streaming-job) Fix NPE in StreamingInsertJob.replayOnCommitted during EditLog replay #62416).
• Fix Broker Load storage properties not being correctly rebuilt after Gson Replay ([fix](load) rebuild broker load storage properties after Gson replay #63094).
• Fix loss of INSERT job statistics in SHOW LOAD after an FE restart ([fix](insert) fix INSERT job statistics lost in show load after FE restart #62331).

Routine Load / Parsing

• Fix an NPE in Routine Load Kafka Meta requests ([fix](job) fix NPE in routine load Kafka meta request #63180).
• Fix an IllegalMonitorStateException in Routine Load when the coordinating BE restarts (branch-4.0: [fix](transaction) fix IllegalMonitorStateException in routine load afterAborted when coordinate BE restarts #62892).
• Fix incorrect column parsing when a UTF-8 BOM CSV uses enclose (branch-4.0: [fix](csv reader) fix incorrect column parsing when using enclose for CSV files with UTF-8 BOM #62092).
• Fix an NPE and load-availability issues when Group Commit selects a Backend ([fix](cloud) Fix NPE in group commit when backend belongs to a different cluster #60652, [fix](group commit) group commit select backend should consider isLoadAvailable #61555).
• Fix conflict validation between load properties and table properties for Broker Load / Routine Load ([fix](BrokerLoad && RoutineLoad) Fix the load attributes and table attributes conflict check #58054).
• Fix to_load_error_http_path returning an incorrect URL in HTTPS mode ([fix](https) fix be to_load_error_http_path return url in https mode #61785).

Storage & Compaction

• Fix BE crashes caused by Schema Cache concurrency issues related to OlapScanner ([fix](olap) Avoid polluting the schema cache in OlapScanner #61510, [refactor](scan) remove SchemaCache to fix concurrent crash in OlapScanner::prepare #62327).
• Fix an IOContext use-after-free crash ([fix](storage) fix IOContext Use-After-Free #59947).
• Fix reading older DecimalV2 Segments that are missing precision / frac ([fix](be) Handle legacy DecimalV2 segments with missing precision/frac #63569).
• Fix blocking during async close polling of Packed Files ([fix](be) Poll packed file async close without blocking #62938).
• Fix an OpenMP concurrency-budget issue when building ANN vector indexes ([codex] fix ANN OpenMP build budget and add concurrency test #61313).

Compute-Storage Decoupling (Cloud Mode)

• Fix several correctness issues during Schema Change: delete local Rowsets before add_rowsets, fill Version holes before running, normalize the Rowset graph before Delete Bitmap Capture, and add an SC_COMPACTION_CONFLICT error code to retry failures across V1 Compaction ([fix](cloud) Delete local rowsets before add_rowsets in cloud schema change #62256, [fix](cloud) Fill schema change version holes before running #63443, branch-4.0: [fix](cloud) Normalize SC rowset graph before delete bitmap capture #63981, [fix](cloud) Add SC_COMPACTION_CONFLICT error code to retry cross-V1 compaction failures #62272).
• Fix several Cache Warmup issues: lack of Packed File support, no retry after errors, NPEs on cancellation/expiration, and cache cleanup ([fix](cloud) Fix warm up cache not supporting packed file #60375, [Fix](warm up) Fix warm up don't retry due to error message change #62886, [fix](cloud) avoid NPE and clear stale cache on warmup job cancel/expire #62805, [fix](cloud) refresh event warmup backends #62839, [fix](cloud) Skip skewed warmup rowset latency samples #62941, [fix](fe) Clear warmup jobs on available backends #62931).
• Fix a balanced_tablets_shards memory leak and a Warmup inflight counting issue ([fix](cloud) Fix balanced_tablets_shards leak memory and fix some file cache case #59093, [fix](cloud) Fix warmup inflight count not decremented on early return #60480).
• Fix Recycler OOM caused by too many queued deletion tasks ([Fix](Recycler) Fix recycler OOM by limiting queued delete tasks #59331).
• Fix a race condition between the first dynamic-partition initialization and a concurrent CREATE MATERIALIZED VIEW ((cloud) Hold table write lock across first-time dynamic partition setup to prevent CREATE MV race #62755).
• Fix exposing the internal KV_TXN_MAYBE_COMMITTED state to clients ([fix](cloud) Hide KV_TXN_MAYBE_COMMITTED from clients #62244).
• Fix Meta Service logs not being printed in cloud mode ([fix](cloud) Fix cloud not print ms log #61766).
• Fix SHOW PROC not showing the Cached Version of partitions ([fix](cloud) show proc should display partition cached version #60807).

Lakehouse & External Data Sources

• Fix loss of predicate filtering when Native and JNI Readers are mixed in FileScanner (branch-4.0: [fix](scan) Fix missing predicate filter when Native and JNI readers are mixed in FileScanner #61802).
• Fix a timezone offset issue for the Hive DATE type in external Readers ([fix](hive) Fix Hive DATE timezone shift in external readers #61330).
• Fix loss of Query TVF column aliases across JDBC Catalogs ([fix](jdbc) Preserve query tvf column aliases across JDBC catalogs #61939).
• Fix incompatibility between SHOW PARTITIONS and the partitions TVF for external Catalogs ([bugfix] fix show partitions command for external catalog validations incompatible with partitions tvf #62134).
• Fix an out-of-bounds crash when generating partition columns for Iceberg / Paimon tables ([fix](scan) Fix OOB crash in partition column generation for Iceberg/Paimon tables #62177).
• Fix several Parquet read/write issues: int96 Timestamp writing, Page V2 encoding, and conditional checks ([fix](parquet)fix parquet write timestamp int96 type. (2/2) #61832, branch-4.0:[fix](parquet)fix parquet write timestamp int96 type. (1/2). #63779, [fix](parquet) Fix wrong encoding for parquet page v2 #63305, [fix](parquet) Fix wrong condition #63509).
• Fix preloading Jindo for the Paimon Scanner (branch-4.0: [fix](fe) Preload paimon jindo for paimon scanner #62351).
• Fix a TVF returning an error when the Thrift message exceeds the size limit ([fix](tvf) fix tvf return error since thrift message too large to reach limit #61788).

File Cache & IO

• Fix a SIGSEGV crash caused by background LRU updates during cache cleanup ([fix](filecache) avoid SIGSEGV in background LRU update when clear cache #60533).
• Fix temporary TTL expiration time being incorrectly anchored to the Tablet creation time (branch-4.0: [fix](filecache) anchor temporary TTL expiration to tablet creation time #62287).
• Fix the file handle cache Key not including the HDFS connection, which could cause connection mismatches ([fix](be) Include HDFS connection in file handle cache key #63516).

Metadata & Job System

• Fix CANCEL ALTER with an empty job list being mistakenly treated as canceling all Rollup jobs ([fix](sync) Treat empty cancel alter job list as all rollup jobs #62712).
• Fix a sessionVariables null pointer after an upgrade ([fix](fe) Fix null pointer exception in sessionVariables after upgrade #61959).
• Fix RestoreCommand throwing an UnsupportedOperationException ([fix](backup) Fix UnsupportedOperationException in RestoreCommand by copying properties before modification #61890).
• Fix Binlog files still being linked when Binlog is not enabled ([fix](snapshot) skip link binlog files if binlog is not enabled #61949).
• Fix auto-partition tables with retention_count not being correctly registered with the scheduler after a restart ([fix](auto-partition) Register auto-partition tables with retention_count in DynamicPartitionScheduler on restart #61954).
• Fix a Host mismatch when starting FE in metadata_failure_recovery mode ([fix](fe) fix host not match if start fe in metadata_failure_recovery #62748).
• Fix being unable to run SHOW TABLET when no database is selected ([fix](fe) Allow show tablet without selected database #63280).
• Fix inconsistent column ordering in SHOW BACKENDS ([fix](fe) Fix SHOW BACKENDS field order mismatch #62207).
• Fix system tables returning unknown statistics ([fix](fe) Return unknown stats for system tables #62913).
• Fix Audit Log issues: UnboundAlias digest calculation, and internal query failures being mislabeled as ERR ([fix](audit) Fix UnboundAlias digest when name comes from child #62160, branch-4.0: [fix](audit) Mark internal query failures as ERR in audit log #62997).
• Fix fe.out not being generated when using start_fe.sh --console ([bugfix](k8s) fix fe.out not generated when start_fe.sh --console #61807).
• Fix the error message for INSERT OVERWRITE ([fix](errmsg) Fix wrong error message of insert overwrite #62555).

Security & Authentication

• Fix Ranger column-level privileges being bypassed when combined with CTEs ([fix](fe) Fix Ranger column-level privilege bypass when CTE combined #61741).
• Fix client IP authentication for Arrow Flight ([fix](auth)Fix arrow flight client ip auth #63506).
• Redact sensitive Headers in Stream Load logs ([fix](fe) Mask sensitive headers in stream load logs #62108).

Stability & Misc

• Fix BE crashes caused by Runtime Filters with a shared Hash table ([fix](be) Fix runtime filter crash with shared hash table #63257).
• Fix a core dump caused by the inline static empty message in arrow::Status ([Fix](arrow flight) Fix arrow::Status inline static empty msg core #63191).
• Fix Arrow UTF8 / String size limit issues ([fix](be) Correct Arrow UTF8/String size limit #63137).
• Fix PartitionRebalancer generating illegal migrations to BEs that lack the required storage medium ([fix](balance) Fix PartitionRebalancer generating invalid moves to BEs without required storage medium #62206).
• Fix jetty_server_max_http_header_size not taking effect under Jetty 12 ([fix](http) Fix jetty_server_max_http_header_size not applied in Jetty 12 #61197).
• Fix Prepared Statement QPS metrics not being counted when Audit Log is disabled ([fix](metrics) Fix prepared statement QPS metrics not counted when audit log disabled #61621).
• Fix the partition-near-limit metric by changing it from a Counter to a Gauge ([fix](metric) Change partition near-limit metrics from counters to gauges #61845).
• Fix loading of the JNI log4j2 configuration ([fix](be) Load JNI log4j2 properties config #63063).
• Track IO-layer read buffers through MemTrackerLimiter to improve memory observability (branch-4.0: [fix](memory) track IO layer read buffers via MemTrackerLimiter using PODArray #62288).

These notes focus on user-visible and operations-visible changes, and omit some purely-internal refactoring and fixes with no external symptoms. For the complete commit history, refer to the PR list for 4.0.6 in the apache/doris repository on GitHub.