Visitar URL original
OPTIMIZE on a Glue Iceberg table can drop a commit that lands while it is publishing · Issue #2733 · Altinity/ClickHouse · GitHub
Skip to content

OPTIMIZE on a Glue Iceberg table can drop a commit that lands while it is publishing #2733

Open

Description

@Selfeer

Describe the bug

OPTIMIZE on a Glue Iceberg table can drop a commit that lands while it is publishing.

Bin-pack compaction re-reads Glue's metadata_location and aborts if that file changed during the rewrite. It then writes the new metadata and calls catalog->updateMetadata. Glue's UpdateTable replaces metadata_location and does not check the snapshot. Any commit that lands after that re-read and before UpdateTable returns is no longer the current snapshot.

This is specific to Glue. A REST catalog rejects the update with assert-ref-snapshot-id and does not move the catalog pointer.

To Reproduce

Needs a Glue Iceberg table with at least two small data files, so OPTIMIZE rewrites them. The insert has to land after OPTIMIZE has rewritten the files and re-read metadata_location, and before Glue UpdateTable returns.

  1. Session A:
OPTIMIZE TABLE glue_db.events
SETTINGS allow_experimental_iceberg_compaction = 1;
  1. Session B, in that window:
INSERT INTO glue_db.events VALUES (1, 'new');
  1. After session A finishes, read the table through Glue:
SELECT count() FROM glue_db.events WHERE note = 'new';

Expected behavior

The count is 1. The insert remains in the current snapshot.

Additional context

Related: #2420

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

antalyaantalya-26.8Session label (releasy session config)

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions