Visitar URL original
datalake_fdw: merge-on-read and DML · Issue #2020 · apache/cloudberry · GitHub
Skip to content

datalake_fdw: merge-on-read and DML #2020

Description

@MisterRaindrop

Part of #2008. Letters (A, B0–B7, C, D, E) are the PRs listed there; this is E.

Scope

  • Merge-on-read: WrapPositionDeleteFilter implemented on the hidden row-ordinal column; position deletes applied while scanning, equality deletes after.
  • DELETE and UPDATE: rows identified by (file, position) through the scan, written as position delete files; UPDATE is delete plus insert through C's writer; one snapshot per statement.
  • A commit that loses the race is retried or fails with a serialization error, never silently dropped.
  • VACUUM stays the current no-op; compaction and snapshot expiry are a follow-up issue.

Out of scope

Copy-on-write, partition evolution, branches and tags (#1683 §2.3).

Depends on

C, D.

Acceptance

  • Deletes made in Cloudberry are seen by Spark and the reverse.
  • UPDATE/DELETE counts match; a rolled-back statement changes nothing visible.
  • Two concurrent transactions on one table: one commits, the other fails or retries; no lost update.
  • A scan over many small delete files stays within the tracked memory budget.

Metadata

Metadata

Assignees

No one assigned

    Labels

    datalakecontrib/datalake_fdw and contrib/datalake_agent: Iceberg lake tables

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions