)

This pull request implements a general limit pushdown optimization for
storage-layer scans on DUP_KEYS and UNIQUE_KEYS (with Merge-on-Write,
MOW) tables. The optimization ensures that LIMIT clauses are enforced
after filtering, preventing the scan from returning fewer rows than
requested when filters are present. The change is carefully scoped to
avoid interfering with existing optimizations (like TopN or storage
merge paths) and includes comprehensive regression tests to validate
correctness across various scenarios.

**General Limit Pushdown Implementation:**

* Added a `general_read_limit` parameter to the storage layer
(`TabletReader`, `RowsetReaderContext`, and `VCollectIterator`) to
enforce a post-filter row limit in non-merge scan paths for DUP_KEYS and
UNIQUE_KEYS with MOW. The iterator now tracks the number of rows
returned and stops reading once the limit is reached, ensuring correct
LIMIT semantics even in the presence of filters.
[[1]](diffhunk://#diff-4d3f2b2a64e1dac3bc076994d3cd9a6c569f7a8c6ad5bc69425188ffccb4266bR494-R503)
[[2]](diffhunk://#diff-4d3f2b2a64e1dac3bc076994d3cd9a6c569f7a8c6ad5bc69425188ffccb4266bR518-R531)
[[3]](diffhunk://#diff-88c1391d53c67a5ab8475c103bb743a2a80cc282e0c752e8fc52b7aaeb2e0e35L95-R102)
[[4]](diffhunk://#diff-88c1391d53c67a5ab8475c103bb743a2a80cc282e0c752e8fc52b7aaeb2e0e35R258-R294)
[[5]](diffhunk://#diff-b7a74d17e270719c904e8c31fb1308cb82ec0b89459a245dce9cb5505021ac63R353-R358)
[[6]](diffhunk://#diff-0025ea2b49c400923787cd77990575f94ba5d925efc95cb16336fbaec92ebfb1R106-R109)
[[7]](diffhunk://#diff-979b99e23ca7a4bb73f5beaa00270494778545296929a3de64b2e87dc8299cd6R202-R203)
[[8]](diffhunk://#diff-48c14ad6bacc0a0ed497ad0cde34756af85aa98dcb7dc909609a288a503be0c9R216-R220)

**Testing and Validation:**

* Added a new regression test suite
(`test_general_limit_pushdown.groovy`) covering DUP_KEYS, UNIQUE_KEYS
(MOW), AGG_KEYS, and MOR tables. The tests verify correct row counts and
output for LIMIT queries with and without filters, as well as negative
cases where the optimization must not apply.
* Added corresponding expected output results for the new tests.

(cherry picked from commit afb1601)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…he#62222)

This pull request introduces significant improvements to scan operator
logic, particularly enhancing the correctness and efficiency of LIMIT
and predicate pushdown in OLAP scans. The main changes include a robust
mechanism for sharing LIMIT counters among scanners, more precise
control over predicate pushdown, and stricter validation for residual
predicates. These changes help ensure that queries with LIMIT and TopN
semantics return accurate results and avoid incorrect or inefficient
execution paths.

**Improvements to LIMIT handling and scanner coordination:**

- Introduced a shared atomic counter (`_shared_scan_limit`) for LIMIT
queries, allowing all scanners to collectively respect the global row
limit. Scanners now check and update this counter to stop early when the
LIMIT is reached, preventing over-scanning and improving efficiency.
TopN scans bypass this mechanism as required.
[[1]](diffhunk://#diff-7fe2376680f9ab835053a0a93c7f9458e6d526acfc180318be0ad936e0c2dedeR50-R55)
[[2]](diffhunk://#diff-7fe2376680f9ab835053a0a93c7f9458e6d526acfc180318be0ad936e0c2dedeR135-R143)
[[3]](diffhunk://#diff-7fe2376680f9ab835053a0a93c7f9458e6d526acfc180318be0ad936e0c2dedeR177-R183)
[[4]](diffhunk://#diff-7fe2376680f9ab835053a0a93c7f9458e6d526acfc180318be0ad936e0c2dedeR196)
[[5]](diffhunk://#diff-9c777f60e1d076777f042a0b702967bd5864bcfe9ee1eee9b456bb67c6fdf817R266-R269)
[[6]](diffhunk://#diff-a45108106b12759815ac5991d56a9d6ead7ffb9c915fb0cf7b6f0b8123b30262R71)
[[7]](diffhunk://#diff-a45108106b12759815ac5991d56a9d6ead7ffb9c915fb0cf7b6f0b8123b30262R239)
[[8]](diffhunk://#diff-a45108106b12759815ac5991d56a9d6ead7ffb9c915fb0cf7b6f0b8123b30262L283-R285)
[[9]](diffhunk://#diff-3c4794a864169735a628d8fcb1de986523a9516aad318aa0a893913f4ce031d4R1043-R1050)

- Adjusted scanner and scan local state interfaces to support the new
shared LIMIT mechanism, including new virtual methods and atomic counter
management.
[[1]](diffhunk://#diff-a45108106b12759815ac5991d56a9d6ead7ffb9c915fb0cf7b6f0b8123b30262R71)
[[2]](diffhunk://#diff-a45108106b12759815ac5991d56a9d6ead7ffb9c915fb0cf7b6f0b8123b30262R239)
[[3]](diffhunk://#diff-a45108106b12759815ac5991d56a9d6ead7ffb9c915fb0cf7b6f0b8123b30262L283-R285)

**Predicate pushdown and validation enhancements:**

- Refactored `_should_push_down_common_expr` to require the expression
as an argument and added logic to ensure only eligible expressions are
pushed down, considering storage merge requirements and key columns.
[[1]](diffhunk://#diff-780cdfbcbb1d30adb63c61521e6b681cad347bebeaa5332a781d05bb51543527L46-R46)
[[2]](diffhunk://#diff-235cc51f4698ebf0ebe796e7253912907bb7c1caf4efcac92538ac1ff2eac171L104-R107)
[[3]](diffhunk://#diff-3ddc75656071d9c0e6b0be450e152a1c94559f7e70ea820e7f0c80a7078e3292L465-R547)
[[4]](diffhunk://#diff-3c4794a864169735a628d8fcb1de986523a9516aad318aa0a893913f4ce031d4L313-R313)

- Added a validation step (`validate_residual_scan_conjuncts`) to
prevent unsupported residual predicates (like SEARCH or disabled MATCH)
and to ensure correctness when using COUNT_ON_INDEX pushdown.

**OLAP scan and segment limit logic improvements:**

- Improved the logic for pushing down segment limits and TopN
optimizations, ensuring that such optimizations are only enabled when
all necessary conditions are met (e.g., no residual predicates, no
runtime filters, storage does not require merging). The code now asserts
correct states and disables shared LIMITs for TopN scans as appropriate.
[[1]](diffhunk://#diff-4d3f2b2a64e1dac3bc076994d3cd9a6c569f7a8c6ad5bc69425188ffccb4266bL446-R502)
[[2]](diffhunk://#diff-3ddc75656071d9c0e6b0be450e152a1c94559f7e70ea820e7f0c80a7078e3292R1155-R1156)

**Code cleanup and maintenance:**

- Removed unused or obsolete logic related to filter block conjuncts and
the old limit quota acquisition method.
[[1]](diffhunk://#diff-4d3f2b2a64e1dac3bc076994d3cd9a6c569f7a8c6ad5bc69425188ffccb4266bL90)
[[2]](diffhunk://#diff-0c9a817d45d8130ea3211189e1321d1275e22fd4a9a3fac2bd707b1cfeefa5e5L115-L135)

**Dependency and include updates:**

- Added missing includes for expression types to support new logic.


```sql
CREATE DATABASE IF NOT EXISTS lm_bench;
USE lm_bench;

DROP TABLE IF EXISTS lm_wide_fact_10m;
CREATE TABLE lm_wide_fact_10m (
    id BIGINT NOT NULL,
    filter_key INT NOT NULL,
    sort_key BIGINT NOT NULL,
    metric BIGINT NOT NULL,
    payload_a VARCHAR(1024) NOT NULL,
    payload_b VARCHAR(1024) NOT NULL,
    long_payload_a VARCHAR(4096) NOT NULL,
    long_payload_b VARCHAR(4096) NOT NULL
)
ENGINE=OLAP
DUPLICATE KEY(`id`)
DISTRIBUTED BY HASH(`id`) BUCKETS 32
PROPERTIES (
    "replication_num" = "1"
);

INSERT INTO lm_wide_fact_10m
SELECT
    number AS id,
    CAST(number % 1000000 AS INT) AS filter_key,
    (number * 2654435761) % 1000000007 AS sort_key,
    (number * 1103515245 + 12345) % 2147483647 AS metric,
    REPEAT(CONCAT('payload_a_', CAST(number AS STRING), '_',
        CAST((number * 1315423911) % 1000000007 AS STRING), '_'), 16) AS payload_a,
    REPEAT(CONCAT('payload_b_', CAST(number AS STRING), '_',
        CAST((number * 2654435761) % 1000000009 AS STRING), '_'), 16) AS payload_b,
    REPEAT(CONCAT('long_payload_a_', CAST(number AS STRING), '_',
        CAST((number * 48271) % 2147483647 AS STRING), '_'), 64) AS long_payload_a,
    REPEAT(CONCAT('long_payload_b_', CAST(number AS STRING), '_',
        CAST((number * 69621) % 2147483629 AS STRING), '_'), 64) AS long_payload_b
FROM numbers("number" = "10000000");

ANALYZE TABLE lm_wide_fact_10m WITH SYNC;


-- ============================================================
-- Case 1: TopN next + LIMIT + high-filter-rate pushed-down predicate
-- Expected observation:
--   filter_key is a narrow pushed-down predicate and keeps about 1% rows.
--   TopN works on narrow metric/sort_key, while long strings are final output.
-- ============================================================

SELECT
    id,
    filter_key,
    metric,
    sort_key,
    payload_a,
    payload_b,
    long_payload_a,
    long_payload_b
FROM lm_wide_fact_10m
WHERE filter_key < 10000
ORDER BY metric DESC, sort_key ASC
LIMIT 10;
-- Result: 0.12 sec -> 0.10 sec

-- ============================================================
-- Case 2: Normal scan + LIMIT
-- Expected observation:
--   No ORDER BY and no predicate. This isolates ordinary scan limit behavior
--   while still returning long string columns.
-- ============================================================

SELECT
    id,
    filter_key,
    metric,
    sort_key,
    payload_a,
    payload_b,
    long_payload_a,
    long_payload_b
FROM lm_wide_fact_10m
LIMIT 10;
-- Result: 0.06 sec -> 0.01 sec

-- ============================================================
-- Case 3: Normal scan + LIMIT + high-filter-rate pushed-down predicate
-- Expected observation:
--   No ORDER BY. This isolates scan predicate + limit behavior with long
--   string result columns and a predicate that filters about 99% of rows.
-- ============================================================

SELECT
    id,
    filter_key,
    metric,
    sort_key,
    payload_a,
    payload_b,
    long_payload_a,
    long_payload_b
FROM lm_wide_fact_10m
WHERE filter_key < 10000
LIMIT 10;
-- Result: 0.07 sec -> 0.02 sec

```

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

### Backport notes (branch-4.2)

Partial pick. This branch keeps the enable_common_expr_pushdown switch and has no
shared scan LIMIT counter (apache#61617 was reverted here by apache#62191), so:

- Not picked: scanner.cpp/.h, scanner_context.cpp/.h and scanner_scheduler.cpp (the
  shared counter), ScanLocalState::shared_scan_limit_ptr(), and the
  collection_statistics_test.cpp hunk (the file does not exist on this branch).
- Not picked: pushing key-only common expressions into SegmentIterator for tables
  merged on read (AGG, UNIQUE without merge-on-write). _should_push_down_common_expr()
  keeps the rule this branch had: switch on, no storage merge, expression acts on a
  slot (VExpr::is_acting_on_a_slot stays). The follow-up fix apache#63279 was only needed by
  that pushdown and is not picked either.
- RuntimeState keeps enable_common_expr_pushdown() and
  enable_common_expr_pushdown_for_inverted_index(). The OlapScanner assertion that
  every slot-based conjunct was pushed down only runs with the switch on. The thrift
  comment on field 107 is unchanged.
- Not picked: validate_residual_scan_conjuncts (the SEARCH / MATCH / COUNT_ON_INDEX
  residual-predicate errors). This branch still has enable_common_expr_pushdown, and
  fuzzy sessions and existing tests such as test_compound_1 turn it off. Those
  predicates then stay in the scanner, which this branch evaluates correctly, so the
  errors would reject valid queries. On master the switch is gone (apache#64285) and the
  tests no longer turn it off (apache#64479).
- Key TopN on tables merged on read keeps the behavior this branch had: without
  runtime filters the scanner sets read_orderby_key* and keeps _limit = sort_limit, so
  it stops after its own N key-ordered rows (master resets _limit to -1 for them).
  Storage TopN (read_orderby_key_limit) is only set for physical DUP_KEYS and
  merge-on-write UNIQUE_KEYS tablets; for a MOR table read as DUP the reader does not
  prepare the key columns that VCollectIterator::_topn_next() compares, so that case
  falls back to reading every row.
- TQueryOptions field id is 231; 219 and 230 are already used on this branch. apache#68785
  gives master the same id.
- SessionVariable keeps the VariableMgr.VarAttr annotation style. The "Legacy session
  variable" comment is not added.
- segment_iterator.cpp: the count fast path check stays ahead of the read_limit EOF
  check, the shrink_char_type_column_suffix_zero() call does not exist here, and
  count_on_index_fastpath's has_read_limit now reads _opts.read_limit (the SNII
  backport apache#68584 pinned it to false because this branch had no read_limit yet).
- segment_iterator_limit_opt_test builds its Schema from (columns, col_ids); this
  branch has no Schema(TabletSchemaSPtr) constructor.

Verified on this branch: BE (ASAN) and FE build with 0 checkstyle violations; the 23
related BE UT suites (183 cases, including SegmentIteratorLimitOptTest) pass; the
regression suites query_p0/limit (18 suites), test_search_slash_in_term,
infer_predicate and test_variant_predefine_index_type pass against the picked .out
files. Without validate_residual_scan_conjuncts, the suites that failed in CI with
residual-predicate errors pass with enable_common_expr_pushdown both off and on, except
test_multi_tokenize_index_not_built and test_search_score_topn_predicates, which fail
with the switch off on this branch's own errors (E-6001, SearchExpr without inverted
index), not on anything from this pick. The kept scanner-level stop was measured on a Release build: on an AGG table
built from 4M input rows, ORDER BY key LIMIT 10 scans 131k rows in 19 ms; the same
binary with master's behavior (_limit reset to -1 for tables merged on read) scans 2.9M
rows in 114 ms. A MOR UNIQUE table goes 98k rows / 20 ms against 2.2M rows / 84 ms.
Results are identical.

(cherry picked from commit 28cc2bf)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

# Conflicts:
#	gensrc/thrift/PaloInternalService.thrift

# Please enter the commit message for your changes. Lines starting
# with '#' will be kept; you may remove them yourself if you want to.
# An empty message aborts the commit.
#
# Author:    Pxl <xl@selectdb.com>
#
# interactive rebase in progress; onto cabcf45
# Last commands done (2 commands done):
#    pick f6d4a8d4292 # [Improvement](scan) update limit pushdown to tablet reader (apache#61713)
#    pick 428d832fcb0 # [Improvement](scan) support push down limit to segment iterator (apache#62222)
# No commands remaining.
# You are currently rebasing branch 'pick-4.2/segment-limit-pushdown' on 'cabcf45c1af'.
#
# Changes to be committed:
#	modified:   be/src/exec/operator/mock_scan_operator.h
#	modified:   be/src/exec/operator/olap_scan_operator.cpp
#	modified:   be/src/exec/operator/olap_scan_operator.h
#	modified:   be/src/exec/operator/scan_operator.cpp
#	modified:   be/src/exec/operator/scan_operator.h
#	modified:   be/src/exec/scan/olap_scanner.cpp
#	modified:   be/src/runtime/runtime_state.h
#	modified:   be/src/storage/iterator/vcollect_iterator.cpp
#	modified:   be/src/storage/iterator/vcollect_iterator.h
#	modified:   be/src/storage/iterators.h
#	modified:   be/src/storage/rowset/beta_rowset_reader.cpp
#	modified:   be/src/storage/rowset/rowset_reader_context.h
#	modified:   be/src/storage/segment/segment_iterator.cpp
#	modified:   be/src/storage/segment/segment_iterator.h
#	modified:   be/src/storage/tablet/tablet_reader.cpp
#	modified:   be/src/storage/tablet/tablet_reader.h
#	new file:   be/test/storage/segment/segment_iterator_limit_opt_test.cpp
#	modified:   fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
#	modified:   gensrc/thrift/PaloInternalService.thrift
#	new file:   regression-test/data/query_p0/limit/test_unified_limit_pushdown.out
#	modified:   regression-test/data/search/test_search_slash_in_term.out
#	modified:   regression-test/data/variant_p0/predefine/test_predefine_type_index.out
#	modified:   regression-test/suites/query_p0/limit/test_general_limit_pushdown.groovy
#	new file:   regression-test/suites/query_p0/limit/test_unified_limit_pushdown.groovy
#	modified:   regression-test/suites/search/test_search_slash_in_term.groovy
#	modified:   regression-test/suites/variant_p0/predefine/test_predefine_type_index.groovy
#
# Untracked files:
#	be/dict/kuromoji/
#	ui/tsconfig.app.tsbuildinfo
#	ui/tsconfig.node.tsbuildinfo
#