Visitar URL original
Add LTTB downsampling as a built-in Table Model table function by DaZuiZui · Pull Request #18725 · apache/iotdb · GitHub
Skip to content

Add LTTB downsampling as a built-in Table Model table function - #18725

Open
DaZuiZui wants to merge 5 commits into
apache:masterfrom
DaZuiZui:feat/lttb-table-function
Open

DaZuiZui wants to merge 5 commits into
apache:masterfrom
DaZuiZui:feat/lttb-table-function

Conversation

@DaZuiZui

@DaZuiZui DaZuiZui commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Description

Closes #18532

Adds LTTB (Largest-Triangle-Three-Buckets) as a built-in Table Model table function for visualization-oriented downsampling, following the conventions of the existing M4 table function.

Function definition

-- Target-count mode
SELECT * FROM LTTB(DATA => t PARTITION BY device ORDER BY time, TIMECOL => 'time', N => 500);

-- Count-window mode
SELECT * FROM LTTB(DATA => t PARTITION BY device ORDER BY time, TIMECOL => 'time', SIZE => 100, SLIDE => 100);

-- Time-window mode
SELECT * FROM LTTB(DATA => t PARTITION BY device ORDER BY time, TIMECOL => 'time',
                   SIZE => 1m, SLIDE => 1m, ORIGIN => 2026-01-01T00:00:00.000+00:00);
  • DATA (set semantics) / TIMECOL / SIZE / SLIDE / ORIGIN reuse the M4 argument semantics and window construction (half-open [start, start+SIZE) windows, ORIGIN alignment, count windows for integer SIZE).
  • N selects target-count mode: the whole partition is buffered and standard LTTB reduces each participant column to exactly N points (all points when M <= N), always preserving the first and last eligible points. Buckets follow [floor(i*(M-2)/(N-2))+1, floor((i+1)*(M-2)/(N-2))+1); the final bucket uses the last point as vertex C. Output uses window_index = 0.
  • SIZE selects window/bucket mode: each window is one LTTB bucket. Per participant column the anchor is chained across windows (the first eligible point seeds it), vertex C is the average point of the next window that carries data for the column (or the window's own last eligible point when there is none), and empty buckets leave the anchor untouched.
  • Participant columns are inferred automatically (INT32/INT64/FLOAT/DOUBLE; other types are rejected with the shared "not allowed columns" message). NULL/non-finite values are ignored per column; columns with no eligible points emit NULL. Area ties resolve to the earliest timestamp (stable input order). Integral columns keep exact long values in the output.
  • Output schema follows M4: window_start, window_end | window_index, partition columns, then <col>_time, <col> per participant column, aligned by output position and padded with NULL.

Validation

Rejected at analysis time: neither/both of N and SIZE; N < 3; N given as a duration; N with SLIDE/ORIGIN; SLIDE/ORIGIN without SIZE; ORIGIN in count-window mode; SLIDE mode differing from SIZE; descending ORDER BY; non-TIMESTAMP TIMECOL; unsupported participant types. All new messages are i18n constants in both en and zh CommonMessages.

Execution and distribution

  • TableDistributedPlanGenerator.canSplitTableFunctionProcessor now excludes LTTB (like FFT), so the whole partition is gathered and ordered (MergeSort over exchanges) before the function runs — fragment-local LTTB samples are not mergeable.
  • Both modes share a 64 MiB per-processor budget for point buffers across all columns; target-count selection indexes and array-growth peaks are reserved before allocation. The per-column MAX_BUFFERED_POINTS cap also remains in effect. Pending windows waiting for a non-empty successor per column share the same budget.
  • Triangle calculations subtract timestamps before converting to double, preserving nanosecond differences at modern epoch timestamps. Window lookahead links each column to its next non-empty closed bucket and emits completed windows in order.

Alternatives considered

  • Fully independent windows in SIZE mode (no chained anchor) would simplify overlapping windows but diverges from the issue's "previously selected point / next-bucket average" definition; chained anchors were chosen.
  • Reusing M4TableFunction's ValueOperator was rejected because LTTB requires numeric values for triangle areas.

This PR has:

  • been self-reviewed.
  • added Javadocs for most classes and all non-trivial methods.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit and planner tests (LTTBTableFunctionTest, LTTBWindowMemoryTest, LTTBRegressionTest, TableFunctionTest). The 18 LTTB unit tests pass in both en and zh; the 24 planner tests pass in en. Memory regression suites also pass individually with a 128 MiB heap.
  • added integration tests (IoTDBLTTBTableFunctionIT, all four tests pass with both -PTableSimpleIT and -PTableClusterIT).

Key changed/added classes (or packages if there are too many classes) in this PR
  • org.apache.iotdb.commons.udf.builtin.relational.tvf.LTTBTableFunction (new)
  • org.apache.iotdb.commons.queryengine.plan.relational.function.TableBuiltinTableFunction
  • org.apache.iotdb.db.queryengine.plan.relational.analyzer.StatementAnalyzer (tryAppendLTTBModeArgument)
  • org.apache.iotdb.db.queryengine.plan.relational.planner.RelationPlanner
  • org.apache.iotdb.db.queryengine.plan.relational.planner.distribute.TableDistributedPlanGenerator
  • org.apache.iotdb.commons.i18n.CommonMessages (en/zh)

🤖 Generated with Claude Code

Recompute a bucket with exact arithmetic when floating-point area or
lookahead calculations overflow, preserving finite peaks and stable ties.
Keep the existing fast path for ordinary input values.

Check count-window ends only when their start is reached, reject an
unrepresentable end, and stop advancing before the next start overflows.
This prevents stalled processing and spurious windows for large sizes.

Add six numeric boundary regressions covering all three sampling modes,
lookahead sums, opposite-sign differences, area ties, and count bounds.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature request] Add LTTB downsampling as a built-in table function for the Table Model

1 participant