A refused DROP INDEX leaves the index half dropped: it answers 0 rows for every record in the buckets it did reach, across a restart, and with no sub-index left it wedges every later DDL #10589

statements are not atomic: a DROP TYPE or ALTER TYPE ... BUCKET - refused midway leaves the
in-memory schema changed") and #10317 ("A refused schema statement still changes the schema") are
the same defect class and both are closed by the fix on main. Neither covers DROP INDEX:
the fix added the up-front probe to dropType, removeSuperType and removeBucket only, and the
test it shipped (Issue10421SchemaStatementsAreAtomicTest) enumerates DROP TYPE (document, vertex,
lock contention), bucket removal, super-type removal and rename. #9848 and #9615 are about a
CONCURRENT drop, not a refused one.

Summary

The rule #10421 installed: a schema statement that is refused changes nothing. It is enforced by
probing, before the first change, every file the statement is going to delete
(LocalSchema.requireFilesCanBeDeleted) and taking the locks of those files once up front. The
probe has exactly three call sites: LocalSchema.dropType, LocalDocumentType.removeSuperType and
LocalDocumentType.removeBucket.

LocalSchema.dropIndexInternal is the fourth site with the same shape and it did not get the probe.
Dropping a TypeIndex is TypeIndex.drop() walking its bucket sub-indexes and calling back into
dropIndexInternal once per sub-index, each deleting one file. It takes the file locks up front
(executeLockingFiles(index.getFileIds(), ...)), which is half of what #10421 asks for, but nothing
asks whether those files CAN be deleted. The first file whose delete fails aborts the statement with
every sub-index before it already dropped, unregistered and saved.

What is left is not a schema anybody meant to reach:

• the TypeIndex wrapper stays registered on the type and existsIndex still answers true, but it
  now covers a SUBSET of the type's buckets. Every record in a bucket whose sub-index was dropped is
  invisible to the index. After a reopen SELECT count(*) FROM T WHERE p = 'v0' answers 0 for
  those records while the record is still there.
• when the failing file happens to be the LAST one, the wrapper is left with ZERO sub-indexes, which
  dropIndexInternal's own comment says must never happen ("A TypeIndex with no remaining bucket
  children must not stay in indexesByProperties: schema serialization (toJSON) calls
  TypeIndex.getPropertyNames(), which fails on an empty wrapper"). The next DDL of any kind then
  fails with IndexException: Index 'T[p]' is not valid. Probably has been drop or rebuilt, so the
  database takes no further schema change at all until it is reopened.
• the damage is persisted, not only in memory: on reopen the schema logs
  Cannot find indexes [T_0_..., T_1_..., T_2_...] defined in type 'T'. Ignoring them and carries
  on with the mutilated index.

The refusal message also does not carry the Nothing was changed wording the other three statements
now end with, which is the one signal a caller has that the statement was clean.

Where (HEAD 8b93bb7)

• engine/src/main/java/com/arcadedb/schema/LocalSchema.java
  • dropIndexInternal(String) (~line 2247): database.executeLockingFiles(index.getFileIds(), ...)
    with no requireFilesCanBeDeleted before it.
  • requireFilesCanBeDeleted(String, Collection<Integer>) (~line 3209): the probe, and its three
    call sites - dropType (~line 3064), LocalDocumentType.removeSuperType (~line 557),
    LocalDocumentType.removeBucket (~line 2764).
• engine/src/test/java/com/arcadedb/schema/Issue10421SchemaStatementsAreAtomicTest.java: the
  enumeration the fix shipped, which has no DROP INDEX case.

Repro

Embedded, eclipse-temurin 25. The fault is the one that test injects: the file
of one bucket sub-index is moved aside and a non-empty directory is put at its path, which
Files.delete refuses.

CREATE DOCUMENT TYPE T BUCKETS 4;
CREATE PROPERTY T.p STRING;
CREATE INDEX ON T (p) NOTUNIQUE;
-- 12 records, p = 'v0' .. 'v11'
-- make the file of sub-index [1] of 4 undeletable, then:
DROP INDEX `T[p]`;

Fault on sub-index 1 of 4:

DROP INDEX -> refused SchemaException
left: wrapper still registered, 2 of 4 sub-indexes, existsIndex=true
a later DDL -> OK
after reopen: 1 index, 2 sub-indexes
  SELECT count(*) FROM T WHERE p = 'v0'  -> 0      (record present)
  same for v1, v4, v5, v8, v9                      6 of 12 records lost to the index

Fault on sub-index 3 of 4 (the last one reached):

DROP INDEX -> refused SchemaException
left: wrapper still registered, 0 sub-indexes, existsIndex=true
CREATE DOCUMENT TYPE Z -> IndexException: Index 'T[p]' is not valid. Probably has been drop or rebuilt
after reopen: WARNING Cannot find indexes [T_0_..., T_1_..., T_2_...] defined in type 'T'. Ignoring them
  9 of 12 records answer 0 through the index

Every statement that reaches dropIndexInternal inherits it. Same setup, fault on sub-index 1 of 4:

ALTER TYPE <edge> WITH unique = false (LocalEdgeType, ~line 256) and
LocalDocumentType.dropSubIndexesNoLongerCovered reach the same method; not injected, same code
path.

Suggested fix

The one-line version is requireFilesCanBeDeleted("drop the index '" + indexName + "'", index.getFileIds())
immediately before the executeLockingFiles in dropIndexInternal, which is exactly what the other
three statements do.

The guarantee worth asking for is the one that would have caught this site when #10421 was written:
make the probe impossible to skip rather than easy to remember. Every statement that deletes
component files should go through one helper that takes the file ids, probes them, locks them and
then runs the mutation - executeLockingFiles and requireFilesCanBeDeleted fused into a single
dropFiles(statement, fileIds, mutation) - so that reaching ComponentFile deletion without the
probe is not expressible. A test that walks every Schema method that can delete a file and asserts
the schema is byte-identical after a refusal would hold the set; today
Issue10421SchemaStatementsAreAtomicTest enumerates five statements by hand and DROP INDEX is
outside the list.

A half-dropped index should also not be allowed to answer queries: a TypeIndex whose sub-index
count no longer matches its type's bucket count is not a valid index, and either the drop completes
or the wrapper is taken down with it.

Scope / confidence

Reproduced on 8b93bb7ee2, embedded, for all four statements in the table, with the fault on the
middle sub-index and on the last one, and across a close/reopen. The fault model is the one the fix
itself chose; any real cause of a failing Files.delete (permissions, a read-only mount, an NFS
ESTALE, a file held open) reaches the same place. Not checked on a replicated database, where the
refusal also has to be reconciled with what the schema session shipped.

Still present at the current main revision: every code point named above was re-checked there before this was filed.