CAS: clean restart cannot attach a table whose live ref names a missing manifest #2838

✅ I checked the Altinity Stable Builds lifecycle table, and the Altinity Stable Build version I'm using is still supported.

Type of problem

Bug report - something's broken

Describe the situation

After a clean restart, ClickHouse cannot attach a ReplicatedMergeTree table on a content-addressed disk. Startup reads a committed ref whose manifest object is gone and refuses to load the table.

Found on branch antalya-26.8, version 26.8.34.20001.altinityantalya, release build. The table had just answered a checksum query. The next step was docker restart of the server. On the way back up, ATTACH failed and later queries sat on the failed load job.

This issue:

• Fails table startup with Code: 107 FILE_DOESNT_EXIST (INV-NO-DANGLE)
• Leaves the table unloaded: queries then fail with Code: 722 / Code: 696 / Code: 695 wrapping that Code: 107
• Uses the current manifest layout (store/<uuid>@cas@/...) and the current CasManifestReader::readManifestShared contract, which still requires this throw

Closed #2052 is the same invariant, triggered by the orphan-manifest sweep, and it was fixed in July 2026. No open issue or pull request covers a restart that cannot attach a table because a live ref names a missing manifest. #2031 item CAS-022 is the other window: a manifest body written before its catalog row.

How to reproduce the behavior

Environment

• Version: 26.8.34.20001.altinityantalya
• Branch: antalya-26.8
• Build type: release
• Topology: two replicas, shared S3 pool, gc_interval_sec 10
• Policy ca_local3: volume hot is local1 + local2 with max_data_part_size_bytes 4194304, volume cas is the content-addressed disk

Steps

1. Create the table on both replicas.

CREATE TABLE default.s37_mixed
(
    id UInt64,
    payload String
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/s37_mixed', '{replica}')
ORDER BY id
SETTINGS storage_policy = 'ca_local3',
         min_bytes_for_wide_part = 0,
         min_rows_for_wide_part = 0;

2. Insert several parts small enough to land on the hot volume (about 6 inserts of 48 rows, 64 KiB payload), then merge them.

OPTIMIZE TABLE default.s37_mixed FINAL;

3. Read the table. This succeeded.

SELECT count(), sum(cityHash64(payload)) FROM default.s37_mixed;

4. Restart the server (docker restart). Startup runs:

ATTACH TABLE default.s37_mixed UUID '7d1e44f2-3944-45b6-a450-e8bd9e63c558'
(
    `id` UInt64,
    `payload` String
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/s37_mixed', '{replica}')
ORDER BY id
SETTINGS storage_policy = 'ca_local3',
         search_orphaned_parts_disks = 'local',
         min_bytes_for_wide_part = 0,
         min_rows_for_wide_part = 0,
         index_granularity = 8192;

This sequence was observed once on the version above. The server log does not show which component removed the manifest object.

Expected behavior

The restart finishes, default.s37_mixed attaches, and the checksum query returns the same result as before the restart.

Actual behavior

On release builds

ATTACH fails while loading a part. The load job stays failed, so later queries fail too.

Code: 107. DB::Exception: live ref names manifest at soak_pool/cas/manifests/ca_soak_clickhouse1/store/7d1/7d1e44f2-3944-45b6-a450-e8bd9e63c558@cas@/0000000000000002-0000000000000004/000001.zst but its object is missing — INV-NO-DANGLE: Cannot attach table `default`.`s37_mixed` ... (FILE_DOESNT_EXIST)

A query issued after that:

Code: 722. DB::Exception: Waited job failed: Code: 696. DB::Exception: Load job 'startup table default.s37_mixed' -> Code: 695. DB::Exception: Load job 'load table default.s37_mixed' failed: Code: 107. DB::Exception: live ref names manifest at soak_pool/cas/manifests/.../000001.zst but its object is missing — INV-NO-DANGLE. (FILE_DOESNT_EXIST)

Frames from the attach failure:

DB::Cas::CasManifestReader::readManifestShared(DB::Cas::ManifestId const&)
DB::Cas::CachedPartFolderAccess::getView(DB::Cas::PartRefKey const&, DB::Cas::Freshness) const
DB::ContentAddressedMetadataStorage::existsFile(String const&) const
DB::DataPartStorageOnDiskFull::existsFileImpl(String const&) const
DB::VersionMetadataOnDisk::VersionMetadataOnDisk(...)
DB::MergeTreeDataPartBuilder::build()
DB::MergeTreeData::loadDataPart(...)

Root cause analysis

CasManifestReader::readManifestShared GETs the manifest named by a live ref. An empty result throws FILE_DOESNT_EXIST and does not substitute an empty manifest. existsFile calls that read while MergeTreeData::loadDataPart is building the part during ATTACH. The ref for .../0000000000000002-0000000000000004/000001.zst was committed. The object was not there. The table load job fails closed, so the table stays unavailable after the restart.

The log does not identify who deleted the object, or whether the object was never durable. readManifestShared is doing what its contract says. The bug is that a table which was readable before a clean restart comes back with a live ref and no manifest body.

Additional context

• Branch: antalya-26.8
• Version: 26.8.34.20001.altinityantalya
• Related: CAS: orphan-manifest sweep deletes live part manifests → fsck reports dangling=2 after basic CAS replication smoke #2052 (closed; orphan-manifest sweep deleted live manifests). CAS — consolidated static analysis audit findings (tracking) #2031 CAS-022 is a manifest body that exists before its catalog row.