Repository navigation
Replicated apply does not advance PageManager.publicationSequence: a replica read can mix chunks of two commits #10941
Description
Summary
On a replica, the replicated apply of a transaction (TransactionManager.applyChangesPages) writes pages into the read cache without advancing PageManager.publicationSequence. That counter is what the #9369 check uses to detect a commit whose pages are only partly published (PageManager keeps it odd while the pages of one transaction are being published). Because the replicated apply never moves it, a reader on a replica cannot tell that a multi-page record (a chunk chain) is half updated, and can assemble one record from the chunks of two different commits.
Where (origin/main)
engine/src/main/java/com/arcadedb/engine/PageManager.javaaround 1916-1940:publicationSequenceis incremented to odd before and back to even after publishing one transaction's pages (An index lookup misses a key that one transaction deletes and re-creates, for readers that run while it commits: 11 to 23 percent ofpid IN :idsreads lose a row, a scan loses none #9369), and exposed at ~2243 for readers.engine/src/main/java/com/arcadedb/engine/TransactionManager.javaapplyChangesPages(~1497): applies the WAL pages of a replicated transaction and does not take part in that protocol.
How it was found
Found by reading the code while investigating a transient read failure on replicas (a reader walking a growing chunk chain while a replicated apply was writing it). The affected reader recovered on its next attempt, so no wrong result was observed, but nothing in the replica path guarantees that.
Expected
The replicated apply publishes its pages under the same protocol as a local commit (odd/even publicationSequence around the whole transaction), so the #9369 check also protects reads on replicas. A regression test: on a replica, read a multi-page record in a loop while the leader repeatedly rewrites it with a different size; every read returns one of the committed versions, never a mix.
Activity
robfrank commentedon Oct 29, 2026
Additional evidence from a three-node load test (3 writers x 15,000 mixed insert/update/delete ops on node 0 with records that grow into chunk chains, concurrent queries on every node; image built from current main + a feature branch, 2026-10-29):
Scan of bucket N skipped M record(s) that could not be read ... result may be incomplete: 1 to 19 skipped records per follower per run. ASELECT/GROUP BYon a follower therefore silently misses records while a replicated commit is being applied; the same records read fine once the follower caught up.BrokenChunkChainException(LocalBucket.loadMultiPageRecord), 1-3 per run, transient, again only on followers.publicationSequence, so a follower reader cannot detect a half-applied chunk chain.Reproduced in 3 of 4 runs before any mitigation on the reading side.
fix(facets): plan 6 task 12 - ThreeNodesFacetLoadTestIT
fix(#10941): a replicated apply publishes its pages under the publica…
robfrank commentedon Oct 30, 2026
A replicated apply wrote pages into the read cache without moving
PageManager's publication sequence, so a replica could not detect a half-published transaction. The apply now brackets its pages with the publication calls, and the sequence is driven by a count of publications in flight so overlapping ones cannot turn it even early.Still open: The end-to-end two-node reader loop from the issue was not reproduced: the invariant is tested at the engine level only. No follow-up issue.
Fixed by #10989.
❤️ Thanks for helping make ArcadeDB better! ArcadeDB is free and open source, sustained by its community. If it's useful to you or your company, please consider becoming a sponsor to keep development going.
fix(facets): plan 6 task 12 - ThreeNodesFacetLoadTestIT
fix(facets): plan 6 task 12 - ThreeNodesFacetLoadTestIT
fix(facets): plan 6 task 12 - ThreeNodesFacetLoadTestIT
fix(facets): plan 6 task 12 - ThreeNodesFacetLoadTestIT
fix(#10941): a replicated apply publishes its pages under the publica…
fix(facets): plan 6 task 12 - ThreeNodesFacetLoadTestIT