🐛 fix(soft-rw): keep stalled peers from a second hold - #845

SoftReadWriteLock could grant a lock beside a live writer in two ways that involve only cooperating processes of one account. 🐛 A commit that landed far below the head after two committers stalled at adjacent generations, and a stale check that counted a waiter's own pause as time the holder's record stayed unchanged. Both let a second process enter while the first still holds, and the first learns of it only at its next heartbeat. Stacked on #842.

A commit lands a whole window behind the log

A commit refuses a link into a slot that compaction already freed by checking, before and after the link, that the previous generation still exists; #815 added _extends_the_log for the single-stall case. Two committers stalled before their links to adjacent slots N-1 and N, while peers made 66 or more commits so that both slots fell out of the retained window, defeat that check. The lower committer linked its late record into its freed slot and took it back only after its own post-link check failed. In that window the upper committer found gen/<N-1> present, so its commit returned True about 70 generations below HEAD, and advance() admitted it as writer while HEAD still named the real holder.

The stall only has to span the window. Between the pre-link check and the link, _publish creates, writes and fsyncs the temporary record, the one step of a commit that waits for storage, so a paused or frozen process is enough. The generation log and this guard came with #738 and #815.

sequenceDiagram
    participant D as Committer D (slot 2)
    participant E as Committer E (slot 3)
    participant P as Peers
    rect rgb(207, 228, 255)
        D->>D: passes pre-link check, stalls before link
        E->>E: passes pre-link check, stalls before link
        P->>P: commit generations 3 to 73, compaction frees slots 2 and 3
    end
    rect rgb(255, 214, 214)
        D->>D: links late record into slot 2
        E->>E: links into slot 3, reads gen/2: present
        Note over E: commit() returns True, E enters beside the writer at 73
        D->>D: reads gen/1: gone, removes gen/2
    end

A slot comes free once the log is a whole window past it, and HEAD stays, so after the link a commit is now also refused when the newest listed generation or HEAD leads it by a window. The lock consults HEAD as well because an NFS client may serve the listing from its cache. A refused record that peers lead by less than a window stays in place, because removing it would free its slot early for a third stalled committer; the lock still removes a record that has no successor. The cost is one listing and one read of HEAD per commit.

A paused waiter evicts a live holder

A contender judged a holder stale by reading its record and only then sampling the clock, and the ledger stamped an unchanged value with that late sample. Any pause between the read and the sample counted as time the record stayed unchanged. A waiting process stopped by job control or a frozen container for about stale_threshold minus the age of the nonce it last saw (60 to 90 s with the defaults) therefore saw a record "unchanged" although the holder rewrote it several times meanwhile.

On resume, advance() marked the holder stale, committed the next generation for the contender and unlinked the holder's record. Both then acted as sole holder until the holder's next heartbeat, up to heartbeat_interval (30 s by default), reported evicted. The window is a few microseconds per poll in a single-threaded contender and wider when other threads hold the GIL. The read-then-sample order dates from #738. The eviction rule it broke is the one the concepts page states: a record unchanged for stale_threshold.

sequenceDiagram
    participant H as Holder
    participant C as Contender
    rect rgb(207, 228, 255)
        C->>C: read holders/H: nonce n1, sample clock t
        H->>H: heartbeat: nonce n2
        C->>C: read holders/H: still returns n1
    end
    rect rgb(255, 214, 214)
        Note over C: paused for stale_threshold before sampling the clock
        H->>H: heartbeats: n3, n4
        C->>C: clock now t + 90.25: n1 "unchanged" for 90.25 s
        C->>H: evicts H, commits next generation, unlinks holders/H
    end

The contender now samples time.monotonic() before the read, and the ledger vouches for an unchanged record only up to that moment, so a stall after the fetch adds nothing. A changed record still gets the sample taken after the read: stamping it with the earlier sample would move the error to the other side, where a stall before the fetch backdates the new value and the next poll counts the stall. Both ends now err toward alive. _unchanged_for_its_lease is the one place that computes the stale verdict, so the sweep of unnamed records follows the same rule.