HA: after a simulated power loss, a recovered node returns records with wrong or missing property names (Jepsen LazyFS) #11151

Summary

In the arcadedb-jepsen LazyFS sweep (production mode, so fsync is on), a node that went through a simulated power loss (unfsynced writes dropped + SIGKILL, then restart and TransactionManager recovery) later serves records whose property names no longer resolve to the right values. Values come back as null, as another property's value, or the record isn't found by its indexed key at all.

Nobody notices until that node serves reads. For the leader-routed workloads, that means when it becomes leader after a later leader power loss. It is intermittent and shows up in roughly one out of three LazyFS runs.

Build: 26.11.1-SNAPSHOT (build d6e16a35e0bc99bb588c6844e8a5e6a8a9bed9b0, arcadedata/arcadedb:latest), 5 nodes, Ratis HA, -Darcadedb.server.mode=production.

Observed cases (all from run 41487873067)

1. bank (acctId comes back null, balances intact)

Schema: Account document type with acctId INTEGER and balance INTEGER, 5 rows.

• 17:09:30: n1 loses power. At 17:09:55 it restarts: Started recovery of database 'jepsen' → Recovery of database 'jepsen' completed.
• 17:10:49: leader n4 loses power, and n1 becomes leader.
• 17:10:51: SELECT acctId, balance FROM Account ORDER BY acctId on n1 returns:

{:result [{:acctId nil, :balance 706} {:acctId nil, :balance 738} {:acctId nil, :balance 1058}
          {:acctId nil, :balance 808} {:acctId nil, :balance 1690}], :returned 5}

All 5 records are there and the balances sum to 5000, but acctId is null on every row until the end of the test. Transfers on n1 (UPDATE Account ... WHERE acctId = N inside LET/IF) keep succeeding.

The same signature ({nil N} reads right after the second leader-targeted power loss) appeared in nightly runs 41162560671 and 41267570817.

2. register-follower (val comes back as the value of key)

Schema: Register with key STRING and val INTEGER. After leader n3 loses power, a follower read
SELECT val FROM Register WHERE key = 'r0' returns val = "r0": the string contents of key, not an integer.

3. elle (indexed key not found, acknowledged writes gone)

Schema: ElleReg with k INTEGER (UNIQUE index) and val INTEGER. Every transaction is a single BEGIN ISOLATION REPEATABLE_READ; ... COMMIT RETRY 5 sqlscript, with reads done inside the transaction.

• Key 47 is created at about t=80s and reaches acknowledged value 14. An :ok read at op 984 sees 14.
• Op 985: leader n2 loses power. n1 becomes leader (n1 had been power-killed and recovered earlier in the test).
• Op 1004 (:ok): [[:r 47 nil] [:r 44 nil] [:w 47 19]]. Key 47 is not found, and the next writes start a new history for it (19, 20, …).

Elle reports :lost-update, :G2-item and :G-single-item-realtime on key 47.

Hypothesis (not confirmed in the code)

Property names are stored as dictionary IDs in the records. All three cases fit a node whose name↔ID dictionary (or schema) state after power-loss recovery differs from the one its records were written with:

• an ID that no longer resolves (acctId → null);
• an ID that resolves to a different name (val → the key value);
• a lookup that misses because the indexed property no longer matches (k = 47 → not found).

This would happen if dictionary/schema updates are not covered by the same fsync + WAL recovery guarantees as page writes, or if Raft replay on the recovered node doesn't re-apply them.

The power loss is simulated with LazyFS on both /opt/arcadedb/databases and /opt/arcadedb/ratis-storage: lazyfs::clear-cache drops every unfsynced page, then the process is SIGKILLed.

Artifacts

Each run's jepsen-store-<run-id> artifact contains:

• jepsen-store.tar.gz: per test, history.txt, results.edn, jepsen.log (bank logs the raw server response on a bad read), and n1..n5/console.log (server output across restarts);
• jepsen-reports/: JUnit summary of the sweep.

The failing tests are under store/arcadedb-{bank,elle,register-follower}-lazyfs-local/.

To reproduce, from arcadedb-jepsen:

./fetch-arcadedb-image.sh && (cd docker && docker compose up -d) && docker exec jepsen-control sh /jepsen/docker/setup-ssh.sh
./run-lazyfs-tests.sh 90