Backport #4765 to 8.0.x: Fail over to a recovered database when the active one is inoperable - #4774

Note

Medium Risk
Changes core MultiDb failover timing and active-DB selection on health events and failed commands; behavior shifts from “stay on broken active until manual failback” to automatic switch to any operable endpoint.

Overview
Fixes a case where MultiDbClient could remain pinned to an inoperable active database even after a lower-weight replica became usable again.

Health status changes: When a non-active database reports healthy, the provider now calls failoverIfActiveInoperable() if the active database is still unhealthy—not only when the active DB turns unhealthy. Failed failover attempts from health events still swallow exceptions so the health-check scheduler keeps running, with an updated warn log.

Command path: assertOperability() (used when a command cannot get a connection from the active DB) now attempts an immediate switch to any operable database instead of only throwing temporary/permanent “no healthy database” exceptions when a candidate exists. Switches use tryLock so in-flight failing commands do not block on an ongoing switch.

Docs clarify that recovery can happen automatically via health checks or the next command (any weight), and tweak SwitchReason.HEALTH_CHECK wording.

Tests are updated and expanded for automatic recovery after permanent outage, command-driven failover, grace-period timing, and forceActiveDatabase pinning behavior.

^{Reviewed by Cursor Bugbot for commit eae1b1d. Bugbot is set up for automated code reviews on this repo. Configure here.}