LocalBucket.freeChunkChain zeroes a slot-table entry before it checks the slot still holds a continuation chunk of this chain, and reports the mismatch as a WARNING where the sibling walk of the same chain raises ConcurrentModificationException #10875

SEVERITY: MEDIUM

Summary

A record that shrinks out of its chunk chain has the leftover chunks freed by freeChunkChain. The loop
zeroes the record-table entry of each chunk and only afterwards reads the slot back to confirm it was a NEXT_CHUNK. When it was not, the slot has already been freed and the method logs a WARNING and stops.

The same two conditions, on a walk of the same chain, are treated as a concurrent modification by updateMultiPageRecord: it checks BEFORE writing and throws ConcurrentModificationException so the
transaction retries. So one walker calls the state "retry, somebody changed the chain under me" and the
other calls it "broken chain, carry on", and the second one has already written by the time it decides.

The broken invariant is the one the method's own javadoc states: "Never throws on a chain it cannot walk,
a broken pointer is logged and the walk stops there ... so the chunks left behind are unreachable rather
than dangerous". The chunks left behind are indeed harmless. The slot the walk landed on is not: it was
freed before it was identified.

Consequence when the slot has been reused rather than merely freed: the record occupying it is unlinked
from its page. It disappears from scan(), from count() and from lookupByRID, in a transaction that
commits normally, with a WARNING naming a different RID. Nothing in the record's own bytes is touched, so CHECK DATABASE sees a consistent bucket afterwards.

Where (HEAD 0d9d8ce)

• engine/src/main/java/com/arcadedb/engine/LocalBucket.java:7218 freeChunkChain, the loop at 7222
  • 7234 the slot-table entry is zeroed
  • 7237 to 7241 only now is recordPositionInPage == 0 noticed (slot already free)
  • 7243 to 7249 only now is recordSize[0] != NEXT_CHUNK noticed (slot holds something that is not a
    continuation chunk of this chain)
• the sibling walk, which checks the identical two conditions before any write and raises
  ConcurrentModificationException on each:
  engine/src/main/java/com/arcadedb/engine/LocalBucket.java:6963 to 6976 (messages at 6968 and 6975)
• callers: LocalBucket.java:7102 (updateMultiPageRecord) and LocalBucket.java:7189
  (collapseChunkChainToRecord)

Repro

What I executed, on an unmodified build of HEAD plus one System.out.println added at each of the two
warning sites purely to print which condition fired and where (the printlns were removed and the class
rebuilt before the fault-injection run below; the worktree is clean at HEAD).

Probe: one document type in one bucket, 16 KB bucket pages, 25 to 40 records, 6 threads doing
read-modify-write with retry on NeedRetryException, each write re-sizing the record randomly across the
in-page / placeholder / chunk-chain boundaries, plus 4 threads inserting and deleting records in the same
bucket so freed slots are recycled. Final check: every record's counter equals the number of commits that
incremented it, the content is not torn, and CHECK DATABASE is clean.

Result, five seeds, 6 threads x 700 updates each:

seed 99:    8 x "freeChunkChain reached a slot that is not a chunk of this chain" (slot already free)
seed 1234:  6 x
seed 777:   9 x
seed 4242: 12 x
seed 31337: 9 x
records=40 lostOrTorn=0 unexpectedFailures=0
CHECK totalErrors=0 warnings=[] corruptedRecords=[] danglingPlaceholderPointers=0
      orphanedChunks=0 crossLinkedChunks=0 staleRecordCounters=0

The engine's own line, verbatim, on the unmodified build:

WARNI [LocalBucket] Error on deleting extra part of multi-page record #1:39 after it shrunk

Sample of the instrumented detail, showing the pointer and the target slot:

PROBE freeChunkChain SLOT-ALREADY-FREE rid=#1:39 ptr=1777664 page=868 slot=0
PROBE freeChunkChain SLOT-ALREADY-FREE rid=#1:39 ptr=1777664 page=868 slot=0
PROBE freeChunkChain SLOT-ALREADY-FREE rid=#1:1  ptr=1689600 page=825 slot=0
PROBE freeChunkChain SLOT-ALREADY-FREE rid=#1:1  ptr=1689600 page=825 slot=0

The same pointer reaching the same already-free slot twice says the pointer the walk follows is not a
transient in-flight value: it is what the committed chain head carries.

The identical probe run SINGLE-THREADED (3300 records, 1500 rounds of mixed update/insert/delete, every
record verified byte for byte and CHECK DATABASE clean) never reaches either condition. So this is a
time-of-check / time-of-use window between the read of the chain and the free of its tail, not a
pre-existing corruption.

What I did NOT reproduce: the destructive variant, where the slot the walk lands on has been reused by a
concurrent insert instead of being left free. In my runs the recycled chunk pages were never re-claimed
inside the window. The branch that would then run is LocalBucket.java:7234 followed by 7245, i.e. the
record-table entry is zeroed and the mismatch noticed afterwards.

Refutation attempted

1. Is the state unreachable because the reader validates the chain first? The reader is strict, and that
   is why a chain that is ALREADY dangling cannot get here. Fault injection: I repointed the chain tail of
   a 48 KB record at an unrelated live record's slot (a probe in package com.arcadedb.engine that
   rewrites the next-chunk pointer through getPageToModify), then tried to update the record. The read
   refuses before the update runs:

   BIG head nextChunkPointer=2048
     chunk 1 at 1:0 marker=-3 next=4096
     ...
     chunk 6 at 6:0 marker=-3 next=0
   injecting victim position 12289 as the chain's tail pointer
   VICTIM before shrink: tag=VICTIM
   com.arcadedb.exception.BrokenChunkChainException: Multi-page record #1:0 has a broken chunk chain
     (unexpected marker at chunk 6): the record is corrupted and cannot be read.
     at com.arcadedb.engine.LocalBucket.loadMultiPageRecord(LocalBucket.java:5980)
   

   That closes the "already corrupt on disk" route and leaves exactly the window the probe above shows is
   entered: between the read that validated the chain and the free that walks it again. The engine itself
   agrees that window exists, because updateMultiPageRecord guards the same two conditions on the same
   chain in the same method call, and calls them a concurrent modification.

2. Is it deliberate? The javadoc at LocalBucket.java:7206 to 7216 argues the case for not throwing, and
   its argument is about the chunks LEFT BEHIND, which is sound. It says nothing about the slot the walk
   is standing on, and the write at 7234 is not mentioned. The two branches also share one message string
   that describes neither condition, so an operator cannot tell them apart.

3. Does the page-version check save it? No. getPageToModify(..., false) fetches the newest committed
   image of the chunk page, which this transaction pinned nothing on (the same property A concurrent update of the same placeholder-backed record is silently lost, with no conflict raised #6141 is about).
   A record inserted into that slot by a transaction that committed before this fetch is in the image
   this write modifies, so the commit-time version check passes and the zeroing is published.

4. Does CHECK DATABASE catch the outcome? No. The victim's bytes are untouched and its slot reads as a
   plain delete, which is exactly what a delete leaves. CHECK DATABASE in my runs reported
   totalErrors=0 with the warnings firing.

Suggested fix

Move the write. Everything needed to validate is already read before it:

• read recordSize and require recordPositionInPage != 0 and recordSize[0] == NEXT_CHUNK FIRST, and
  only then zero the slot-table entry and call freedWithoutClaiming;
• give the two refusals distinct messages, and make them say which slot was refused.

Whether the refusal should become the ConcurrentModificationException the sibling raises is a separate
call: throwing would make the transaction retry, which is correct but changes behaviour for a chain that
really is corrupt. Leaving it as a WARNING is defensible once nothing has been written.

The guarantee to ask for: no slot-table entry is ever cleared that has not been proven, from the page
itself, to be a continuation chunk of the chain being freed. It is a reordering of four lines, no storage
format change and no new state.

Scope / confidence

Executed: the concurrent probe (5 seeds, both conditions instrumented), the single-threaded control, and
the fault-injection run that establishes the reader is strict. Read, not executed: the reuse of a freed
chunk slot by a concurrent insert inside the window, which is the step from "a slot that is already free"
to "a slot that holds somebody else's record". The ordering defect itself is in the code unconditionally.

Dedup

Checked #10701 (chunk chain linked to slot 0 of page 0, the 0 end-of-chain sentinel, OPEN), #10853
(encode next-chunk links as -(position+1), OPEN), #10859 and #10669/#10700 (small slot on a sealed page),
#9620 (REPEATABLE_READ refusing the first read of a multi-page record). None of them is about the write
order inside freeChunkChain or about the asymmetry with updateMultiPageRecord's guard. Nothing in the
last 1000 titles matches "freeChunkChain", "shrunk" or "extra part".