Visitar URL original
[#13176] fix(core): Guard non-force parent drops inside transactions by yuqi1129 · Pull Request #13499 · apache/gravitino · GitHub
Skip to content

[#13176] fix(core): Guard non-force parent drops inside transactions - #13499

Draft
yuqi1129 wants to merge 7 commits into
apache:mainfrom
yuqi1129:fix/13176-nonforce-drop-checks
Draft

yuqi1129 wants to merge 7 commits into
apache:mainfrom
yuqi1129:fix/13176-nonforce-drop-checks

Conversation

@yuqi1129

@yuqi1129 yuqi1129 commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

  • Check non-force metalake deletion for newly created catalogs inside the store transaction.
  • Check non-force catalog deletion for newly created schemas inside the store transaction. Preserve the existing handling of built-in, imported, and externally removed schemas by allowing only the schema IDs classified before deletion.
  • Keep the catalog-specific condition in the optional SupportsConditionalCatalogDelete capability; EntityStore and RelationalBackend remain unchanged.
  • Preserve the metalake delete cleanup of metadata left under already deleted catalogs.
  • Preserve the catalog delete cleanup of metadata left under already deleted schemas, including the empty-catalog non-force path.

Why are the changes needed?

A direct child catalog or schema created after the manager's emptiness check could previously be removed by the subsequent cascade delete. The transactional check rejects that deletion instead.

This PR guards direct child catalogs and schemas. It does not yet guard tables or deeper descendants created inside an allowed built-in or imported schema; those retain the existing cascade behavior and require a separate design. The metalake in-use fan-out and force-drop lifecycle also remain follow-up work. Related to #13176.

Does this PR introduce any user-facing change?

Yes. A non-force drop rejects a concurrently created direct child catalog or schema rather than silently deleting it. There are no REST or configuration changes. Custom stores and relational backends can opt into the catalog-specific capability; without it, non-force deletion of a catalog containing classified built-in, imported, or externally removed schema metadata fails closed with an unsupported-operation error that suggests the force option.

How was this patch tested?

  • Targeted TestCatalogManager, TestRelationalEntityStore, TestCatalogMetaService, and TestMetalakeMetaService across H2, MySQL, and PostgreSQL. After correcting the new orphan test's assertion to query the schema row directly, that test passed on all three backends.
  • Regression tests for the no-schema manager path, late-schema rejection and preserved cause, unsupported backend message, and orphaned catalog child cleanup.
  • Latest follow-up: targeted non-force catalog rejection and orphan-cleanup tests passed on H2, MySQL, and PostgreSQL; :core:spotlessApply and git diff --check passed.
  • :api:spotlessApply, :core:spotlessApply, and git diff --check.
  • Started :core:test -PskipITs -PskipDockerTests=true; it hit an unrelated three-minute timeout in TestLocalJobExecutor.testOutputIndexKeepsSpecialCharactersInWorkingDir. The long-running suite was stopped after the targeted checks above passed.

Copilot AI lite review requested due to automatic review settings September 24, 2026 07:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@yuqi1129 yuqi1129 self-assigned this Sep 24, 2026
A non-force metalake drop now takes the non-cascading delete path, which
soft-deleted users, roles, tags, and other metalake-owned rows but not
policies or their versions, leaving them live under a deleted metalake.

Build both paths from one list of metalake-scoped cleanups so they cannot
drift apart again; the cascading path adds the catalog-scoped cleanups.
@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Code Coverage Report

Overall Project 71.0% +0.22% 🟢
Files changed 81.8% 🟢

Module Coverage
aliyun 19.74% 🔴
api 53.58% -0.04% 🟢
authorization-common 85.96% 🟢
authorization-ranger 14.44% 🔴
aws 53.54% 🟢
aws-bundle 0.0% 🔴
azure 32.1% 🔴
azure-bundle 0.0% 🔴
catalog-common 29.49% 🔴
catalog-fileset 82.06% 🟢
catalog-glue 73.24% 🟢
catalog-hive 83.63% 🟢
catalog-jdbc-common 46.31% 🟢
catalog-jdbc-doris 87.56% 🟢
catalog-jdbc-mysql 81.8% 🟢
catalog-jdbc-postgresql 83.89% 🟢
catalog-jdbc-starrocks 79.16% 🟢
catalog-kafka 75.21% 🟢
catalog-lakehouse-generic 78.48% 🟢
catalog-lakehouse-hudi 79.1% 🟢
catalog-lakehouse-iceberg 85.93% 🟢
catalog-lakehouse-paimon 84.36% 🟢
catalog-model 77.99% 🟢
cli 44.62% 🟢
client-java 78.31% 🟢
client-java-runtime 0.0% 🔴
common 60.13% 🟢
core 86.68% -0.35% 🟢
filesystem-hadoop3 76.51% 🟢
flink 55.25% 🟢
flink-common 32.08% 🔴
flink-runtime 0.0% 🔴
gcp 32.2% 🔴
hadoop-auth 68.0% 🟢
hadoop-common 17.84% 🔴
hive-metastore-common 62.3% 🟢
hive-metastore3-libs 58.8% 🟢
iceberg-aliyun-bundle 0.0% 🔴
iceberg-common 66.53% 🟢
iceberg-rest-server 76.97% 🟢
idp-basic 86.77% 🟢
integration-test-common 0.0% 🔴
jobs 70.88% 🟢
lance-common 36.14% 🔴
lance-rest-server 69.51% 🟢
lineage 59.39% 🟢
optimizer 83.94% 🟢
optimizer-api 42.56% 🟢
server 91.11% 🟢
server-common 82.4% 🟢
spark 61.79% 🟢
tencent 81.78% 🟢
trino-connector 62.12% 🟢
Files
Module File Coverage
api NonEmptyCatalogException.java 0.0% 🔴
core MetalakeMetaService.java 100.0% 🟢
CatalogMetaService.java 98.18% 🟢
RelationalEntityStore.java 88.24% 🟢
JDBCBackend.java 81.61% 🟢
MetalakeManager.java 81.45% 🟢
CatalogManager.java 76.39% 🟢
SupportsConditionalCatalogDelete.java 0.0% 🔴

…s and cover the manager path

- Keep UnsupportedOperationException unwrapped in dropCatalog so REST reports an
  unsupported operation instead of an internal error, with an actionable message.
- Add CatalogManager tests for the allowlist delete, a schema created after
  classification, and a store without SupportsConditionalCatalogDelete.
- Make the in-memory test store drop allowed schemas with the catalog.
@yuqi1129
yuqi1129 requested a review from jerryshao September 30, 2026 07:37

@jerryshao jerryshao left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: ship it

Reviewed the full change on the PR head with the repository checked out (apache/gravitino at 520d9ec, diffed against origin/main = ab07853). No earlier Claude review exists on this PR, so this covers the whole change.

The core of the change is sound, and the two load-bearing assumptions hold in the code as it stands:

  • CatalogMetaService.deleteCatalog line 311 takes the catalog row with selectCatalogMetaByIdForUpdate (see deleteCatalogWithVersion, CatalogMetaService.java:521-530), and SchemaMetaService.insertSchema locks the same row first (lockCatalogForSchemaCreate, SchemaMetaService.java:172-177). So the delete-then-check order really does fence a concurrent schema create.
  • CatalogMetaMapper / MetalakeMetaMapper: insertCatalog locks the metalake row before inserting (CatalogMetaService.java:201-202), so the same argument holds one level up for the metalake path.

Two other things I checked and found correct, in case they come up in review:

  • The MetalakeMetaService refactor preserves the cascade cleanup set exactly: the old cascade list had 30 entries, and catalogScopedCleanups (15) + metalakeScopedCleanups (15) is the same 30. SemanticModel* and View* are correctly in the catalog-scoped list, since both are namespaced under (metalake, catalog, schema).
  • JDBCBackend.deleteCatalogWithAllowedSchemas writes the change log unconditionally where delete() gates on shouldRecordEntityDrop. That is equivalent here: CATALOG is in BaseEntityCache.CACHEABLE_TYPES (BaseEntityCache.java:55-66), so the predicate is always true for this entity type.
  • The comment at CatalogManager.java:1483-1485 about REST reporting an unsupported operation is accurate: ExceptionHandlers.BaseExceptionHandler maps UnsupportedOperationException to Utils.unsupportedOperation (ExceptionHandlers.java:1247-1250).

One thing worth stating explicitly, because it is what makes the new UnsupportedOperationException safe: on the non-force path, a managed-storage catalog can only reach the store delete with an empty schemaEntities, because containsUserCreatedSchemas returns true for any non-empty schema list on such a catalog (CatalogManager.java:1525-1530). So the allowlist branch is only ever reached for unmanaged catalogs, where nothing destructive (physical ops.dropSchema, secret deletion) has happened yet when the transaction rolls back. The rejection is therefore side-effect free. This is subtle enough that a short comment near CatalogManager.java:1425 would help the next reader.

Tests: adequate for the paths this PR adds. The new TestCatalogMetaService / TestEntityChangeLogService cases cover both outcomes of the allowlist delete against H2, and the three CatalogManager tests cover the allowlist call, the late-schema rejection and the fail-closed store. Two gaps I would like to see closed, neither blocking:

  • Nothing exercises the schemaEntities.isEmpty() -> store.delete(ident, CATALOG, false) branch (CatalogManager.java:1424) from the manager. That is the branch every managed-storage catalog now takes, and it is the first time CatalogMetaService's non-cascade catalog path is reachable from production code — before this PR CatalogManager always passed cascade = true, so that else branch (CatalogMetaService.java:396-437) was dead outside unit tests. TestCatalogMetaService.testNonCascadeDeleteRollsBackCatalogFence covers the service, but not the manager wiring.
  • No test asserts the NonEmptyCatalogException message/cause for the late-schema case beyond the exception type.

I could not compile or run tests in this environment: only JDK 21 is installed and build.gradle.kts:62-64 rejects anything other than JDK 17, so every finding below comes from reading the code rather than from execution. CI results on the PR should be the source of truth for compilation and the new tests.

Review drafted with Claude, requested by @jerryshao.

Comment thread core/src/main/java/org/apache/gravitino/catalog/CatalogManager.java Outdated
@yuqi1129

Copy link
Copy Markdown
Contributor Author

@jerryshao
Pending this one, as the fix will add the interface SupportsConditionalCatalogDelete and modify the core storage API. If this is adopted, we may need the class SupportsConditionalSchemaDelete when handling schema deletion. We need to enhance the compatibility of the entity storage API before doing such refactoring.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants