Shared state

How it works

Registry and key space. NewSharedStateRegistry wraps one Connector and a clusterKey prefix. Every counter is stored under <clusterKey>/<key> — for example prod-main/v2/latestBlock/<upstreamId>/<sha256-hex> (see Counter families below). The registry holds a sync.Map of live counterInt64 instances; calling GetCounterInt64 with the same key always returns the same object (idempotent LoadOrStore). On first call, two async bootstrap tasks start: one seeds the local counter with a single remote Get, and one calls WatchCounterInt64 to open a long-lived change stream from the connector.

Write path. TryUpdate is synchronous and local-only — it atomically advances the in-memory counter via a CAS loop and then schedules a background push without touching the network. scheduleBackgroundPushCurrent coalesces concurrent pushes: a single goroutine per counter (1) publishes the current state via connector.PublishCounterInt64 for fast peer notification, (2) acquires the distributed lock within lockMaxWait, (3) reads remote state and adopts it if the remote value is newer, then writes local state if local is newer. If the lock cannot be acquired within lockMaxWait, the goroutine falls back to publish-only propagation and reschedules if another update arrived while it was running.

Foreground latency budget. TryUpdateIfStale is used by the EVM state poller to debounce expensive RPC calls. It launches the upstream RPC in a background goroutine and waits at most updateMaxWait for the result. If the RPC finishes within that window the fresh value is returned synchronously; otherwise the stale local value is returned immediately and the RPC completes asynchronously. The foreground path never acquires the distributed lock. One caller may wait longer: a request naming the finalized tag at its idle edge, when fewer than the quorum's voters reported finalized within the freshness window W, waits for its finalized refresh up to W, past updateMaxWait (see block tracking).

Rollback handling. All counter updates enforce a rollback policy. latestBlock and finalizedBlock use ignoreRollbackOf = 1024 — decreases up to 1024 blocks are silently discarded as noise from lagging upstreams; a larger gap applies only once the lower side has moved: a later reading above the first lower one, with no reading within 1024 of the counter in between (admitLargeRollback). A genuine deep reorg or a chain restarted lower keeps moving from its new height, so it applies at its second reading and triggers the OnLargeRollback callback; a provider answering now and then from a node stuck far behind repeats one value, and its readings at the real head in between clear the count, so it never applies. Lower readings alone for 5 minutes (largeRollbackMaxHold) also apply, so a chain that restarted lower and halted is still followed. earliestBlock uses ignoreRollbackOf = 0 — all decreases are accepted because node pruning genuinely shifts the historical anchor.

Timestamp order and conflict resolution. Each counter state is serialised as {"v":<int64>,"t":<unix_ms>,"b":"<instance_id>"}. The t field is the reconcile order key, by the writer's clock; it never decides freshness. allocateUpdatedAtMs ensures local timestamps are strictly monotonic even under concurrent goroutines (a CAS spin-loop). Head counters (latestBlock, finalizedBlock) are max-only on every path (pubsub, initial fetch, reconcile, background push): a higher remote value is taken whatever its timestamp, so a writer whose clock runs ahead cannot freeze the others; timestamps decide only drops larger than ignoreRollbackOf. Other counters (earliestBlock, cordon, SVM floors) adopt a remote state only when st.UpdatedAt > currentTs. When a remote update has a newer timestamp but a value that would imply a small rollback (within ignoreRollbackOf), advanceTimestampPast is called to advance the local order key past the remote's so the local (higher) value wins the next reconciliation cycle.

Transport differences by connector. All five connectors implement the same GetCounterInt64 / PublishCounterInt64 / WatchCounterInt64 interface; propagation latency differs significantly:

• Redis — push via PUBLISH counter:<fullKey> (e.g. counter:prod-main/v2/latestBlock/<upstreamId>/<sha256-hex>) and subscribe via a single shared PSubscribe("counter:*") connection that matches all counters regardless of cluster; subscribers filter by exact key inside runMessageLoop. Reconnects automatically with exponential backoff (1 s → 30 s with jitter) and a 5-minute periodic poll fallback; pollAllKeys() is triggered after each reconnect to close the gap during the reconnection window. Subscriber channels are buffered (size 1, drop-on-full) so slow consumers never block the message loop. Source: data/redis_pubsub_manager.go:L155-460, data/redis.go:L591
• PostgreSQL — push via pg_notify using a channel name derived by sanitizeChannelName(fmt.Sprintf("counter_%s", key)) to strip characters illegal in PostgreSQL NOTIFY channel names; LISTEN-based subscription with a 30 s polling fallback. Source: data/postgresql.go:L620-718
• DynamoDB — no push primitive. PublishCounterInt64 is a documented no-op; propagation relies on polling at statePollInterval (default 5 s). Each poll delivers the stored state whenever it differs from the last one delivered, an older timestamp included: the counter decides. Cross-pod convergence latency is bounded by the poll interval, not a push event. Source: data/dynamodb.go:L715-766
• Memory — both WatchCounterInt64 and PublishCounterInt64 are no-ops; in-process only, no cross-pod coordination. Source: data/memory.go:L195-207
• gRPC — both WatchCounterInt64 and PublishCounterInt64 return an error; the gRPC connector cannot participate in shared state.

Deadlock-prevention invariant. The foreground paths (TryUpdate and TryUpdateIfStale) never acquire the distributed lock. Only the background scheduleBackgroundPushCurrent goroutine acquires it, and only after any local update is complete. This strict lock-ordering rule was introduced to fix a deadlock where the old design could have scheduleBackgroundPushCurrent hold the distributed lock and wait for updateMu, while TryUpdateIfStale held updateMu and waited for the distributed lock. The test TestLockOrderSimulation_OldBuggyBehavior documents the old behavior.

Counter families. These counters are materialised during normal operation, all stored under <clusterKey>/<key> in the backing store:

1. v2/latestBlock/<upstreamKey> — latest block per upstream (ignoreRollbackOf=1024).
2. v2/finalizedBlock/<upstreamKey> — finalized block per upstream (ignoreRollbackOf=1024).
3. v2/earliestBlock/<upstreamKey>/<probeType> — earliest available block per upstream per probe type (ignoreRollbackOf=0; all decreases accepted because node pruning genuinely shifts the historical anchor).
4. v2/operatorCordon/<projectId>/<upstreamKey> — operator cordon state per upstream.
5. network/<projectId>/<networkId>/finalizedBlock — the highest network-wide finalized any instance served (see Finalized floor) (ignoreRollbackOf=1024).

<upstreamKey> is common.UniqueUpstreamKey(up): the upstream id followed by a hash of its id, endpoint and JSON-RPC headers. It does not depend on chain detection, so it stays the same for the upstream's lifetime. Two upstreams share a key only when they call the same endpoint with the same headers, and so report the same chain. Changing an upstream's endpoint or headers starts a new counter, and so does upgrading from a version that also hashed the network id; the old keys are left behind in the store. Source: common/upstream.go:L68-82

Instance ID. Each counter state carries an UpdatedBy field ("b" in JSON) identifying which pod wrote it. It is resolved from the first non-empty value of: env INSTANCE_ID → env POD_NAME → env HOSTNAME → os.Hostname() → "unknown". This is diagnostic-only and does not affect routing. Source: data/shared_state_registry.go:L82-98

SuggestLatestBlock fast path. When the eRPC proxy observes a block number in an upstream response, it immediately calls SuggestLatestBlock(blockNumber) which calls TryUpdate if the new value is greater than the current local value. This avoids a full polling round-trip and immediately triggers background propagation, advancing the shared counter without waiting for the next polling tick. Source: architecture/evm/evm_state_poller.go:L459-481

IsStale semantics. IsStale(d) reads this instance's clock only: it returns true unless the counter's report time (ReportAt: a local report's evidence time, a remote value's receive time) or the end of its last refresh (an answer, an error or the foreground timeout) is within d. It is the gate inside TryUpdateIfStale before attempting a refresh. Source: data/shared_state_variable.go:L133-141

OnValue / OnLargeRollback callbacks. Registered at counter construction time; called synchronously from triggerValueCallback within both processNewValue (local write path) and processNewState (remote sync path) after a successful CAS. The EVM state poller registers OnValue to push the new block number into the health tracker for metrics and OnLargeRollback to record large rollback events. Source: architecture/evm/evm_state_poller.go:L126-139

Rollback write (rewind). When the network-wide chain tracker re-anchors its pins, or moves to a lower head, each upstream whose own hash vote is on the new branch and whose latest counter is above that vote has the counter lowered to the vote's height through EvmStatePoller.RewindLatestBlock → Rewind. A rewind takes any drop, fires OnRewind (the health tracker's per-upstream head follows it) and then OnValue, never OnLargeRollback, takes a fresh UpdatedAt and is published with "r":true. Another instance takes a rewind-marked lower state newer than its own, without pushing back; an unmarked lower state keeps the rules above. While a counter's value comes from a drop (a large rollback or a rewind), a higher state stamped no later than the drop predates it and is rejected (stale_timestamp), on pubsub and in reconcile, until the counter rises again. Source: data/shared_state_variable.go:L637-692

Poll timeout in EVM state poller. The state poller computes its per-tick context timeout as lockTtl + 15s (minimum 30 s) to allow time for acquiring the distributed lock plus network operations. Source: architecture/evm/evm_state_poller.go:L174-187

Bootstrap task retry. If WatchCounterInt64 fails at startup (connector not ready, network error), initCounterSync marks the watch task as failed and the Initializer retries it in the background. During this window the counter operates in local-only mode. Source: data/shared_state_registry.go:L168-222

Served tip. Each instance computes the served tip from the shared per-upstream counters (see Block tracking). The one network-level counter is network/<project>/<network>/finalizedBlock, the highest network-wide finalized any instance served: a floor for finalized only while a voter of the reading instance reports it. No served latest is shared. Per-group served-tip lanes are in-process, not shared state.

Fallback when absent. erpc/erpc.go:L49-L66 synthesises an in-memory registry if sharedState == nil; the same GetCounterInt64 API works identically but no cross-pod propagation occurs. Init failures are non-fatal: erpc/init.go logs a warning and continues with a synthesised in-memory registry. A broken Redis config therefore produces degraded per-pod behavior, not a hard startup failure.

Config schema

All fields are under database.sharedState in erpc.yaml. The block is optional; its absence causes a silent fallback to a process-local in-memory registry. Struct at common/config.go:L269-286, defaults at common/defaults.go:L789-828, validation at common/validation.go:L263-309.

Top-level clusterKey (common/config.go:L40): defaults to "erpc-default". Propagated to database.sharedState.clusterKey if that field is empty.

DynamoDB-specific statePollInterval (common/config.go:L441): default 5s. Controls the polling rate inside WatchCounterInt64 since DynamoDB has no pub/sub primitive.

Worked examples

All patterns below are distilled from real production fleets; comments explain the non-obvious choices.

1. Minimal Redis config with an explicit cluster key (recommended baseline). Every production deployment sets a unique clusterKey so that staging and production instances sharing one Redis instance don't contaminate each other's block-number state. The explicit connector.id avoids the auto-assignment gotcha where shared-state connectors get "redis" instead of "shared-state-redis":

database:  sharedState:    # unique per deployment — prevents cross-pod block-height contamination    # if two clusters share the same Redis instance (e.g. prod + staging)    clusterKey: "prod-us-east-1"    connector:      # always set id explicitly: auto-assigned id here is "redis", not "shared-state-redis"      id: "shared-state-redis"      driver: redis      redis:        uri: "${SHARED_STATE_REDIS_URL}"

2. Multi-region fleet with relaxed lock and wide foreground wait. When pods span multiple regions, the Redis round-trip from any one pod may vary. This production multi-region config raises lockTtl to 30 s to accommodate the longer distributed-lock cycle, sets lockMaxWait to 200 ms to fail fast and fall back to publish-only propagation rather than blocking, and raises updateMaxWait to 2 s to give the foreground refresh more time before falling through to a stale value:

database:  sharedState:    clusterKey: "fly-prod-aggregator-shard0"    # long TTL covers one full Get+Set cycle at cross-region Redis latency;    # must stay > fallbackTimeout (default 3s), so 30s is generous    lockTtl: "30s"    # fail fast on lock contention — proceed with publish-only; never block requests    lockMaxWait: "200ms"    # cross-region Redis RTT can be 50–200ms; give the refresh goroutine time to finish    # synchronously before falling through to the stale value    updateMaxWait: "2s"    connector:      driver: redis      redis:        uri: "redis://erpc-redis.internal:6379"

3. Same Redis, two logical clusters (staging + prod). When you can't provision separate Redis instances, use distinct clusterKey values. Counter keys are prefixed with the cluster key so staging/latestBlock/... and prod/latestBlock/... never collide, even though both clusters publish to the same Redis PUBLISH channel namespace. The fallbackTimeout and lockTtl defaults are left in place since in-cluster Redis latency is well under 3 s:

# staging erpc.yamldatabase:  sharedState:    # distinct from the production clusterKey — same Redis, no cross-contamination    clusterKey: "staging"    connector:      driver: redis      redis:        uri: "${SHARED_STATE_REDIS_URL}"---# production erpc.yamldatabase:  sharedState:    clusterKey: "prod"    connector:      driver: redis      redis:        uri: "${SHARED_STATE_REDIS_URL}"

4. DynamoDB for AWS-only deployments. When Redis or PostgreSQL aren't available, DynamoDB is a valid fallback, but PublishCounterInt64 is a no-op — propagation relies entirely on polling. Cross-pod convergence latency is bounded by statePollInterval (default 5 s) rather than a push event. Acceptable for batch indexing workloads that tolerate slightly stale block-height views, but not recommended for latency-sensitive RPC fleets:

database:  sharedState:    clusterKey: "prod-main"    connector:      driver: dynamodb      dynamodb:        region: "us-east-1"        table: "erpc-shared-state"        # DynamoDB has no pub/sub — all propagation is via polling;        # lower values cost more DynamoDB read capacity; default 5s is the floor        statePollInterval: 5s

5. No shared state (single-pod or dev). Omitting the block entirely gives you a silent per-process in-memory registry — no config needed, same API, no cross-pod propagation:

# No database.sharedState block — eRPC synthesises an in-memory registry automatically
database: {}

Request/response behavior

The shared-state layer has no effect on JSON-RPC request or response shapes visible to clients. Its impact is routing-internal:

• Block-number reads (latestBlock, finalizedBlock) used by the upstream selector and tag-aware routing come from the in-process atomic copy — zero latency on the request path.
• Served-finalized counter (network/<project>/<network>/finalizedBlock) is raised when an instance computes a higher network-wide finalized; the head computation reads the in-process copy and never waits on the store. Other instances serve it only within maxRegressionBlocks of their own tip and while one of their own voters reports it finalized, so a value no voter backs is ignored. Only raises are published.
• Block-number refreshes are debounced via TryUpdateIfStale; the foreground caller waits at most updateMaxWait (default 50 ms). If the refresh completes within that window the fresh value influences the current request; otherwise the stale value is used and the refresh completes asynchronously for the next request. A finalized-tag request at its idle edge is the exception: it waits for the finalized refresh up to W.
• A broken or unreachable shared-state connector degrades to per-pod local state — no error is returned to the client; routing quality degrades silently.
• Operator cordons (erpc_cordonUpstream) are one more shared counter per upstream (operatorCordon, value = cordon time in unix-ms, 0 = lifted), with the same propagation, bootstrap fetch and local-first semantics as the block counters. See Cordoning.

Best practices

• Always set a unique clusterKey per logical deployment (e.g. "prod-v2", "staging"). The default "erpc-default" is shared by every unconfigured instance pointing at the same store — cross-contamination breaks block-height monotonicity.
• Always set connector.id explicitly (id: "shared-state-redis"). The auto-assigned id for shared-state connectors is string(driver) (e.g. "redis"), not the "shared-state-redis" pattern used by cache connectors — copying connector blocks between scopes silently changes the id.
• Prefer Redis or PostgreSQL over DynamoDB for shared state. DynamoDB has no pub/sub; convergence latency is 5 s by default vs. milliseconds for Redis pubsub.
• Do not set fallbackTimeout low to try to speed up request latency — fallbackTimeout governs background I/O only. The foreground latency knob is updateMaxWait.
• Keep lockTtl > fallbackTimeout (the default 4 s > 3 s relationship). Validation enforces this, but understand it: lockTtl must be long enough for one full Get+Set cycle under the lock.
• In AWS deployments with Redis, set pool_size in the Redis URI (?pool_size=10) to avoid connection exhaustion under bursty background push traffic.
• Monitor erpc_upstream_block_head_large_rollback in production. A sustained non-zero value means an upstream is experiencing genuine reorgs or provider resets — something worth routing around. A provider flapping between a healthy node and one stuck far behind no longer sets it: its drops never apply.

Edge cases & gotchas

1. Value=0 is valid, UpdatedAt=0 is not. Genesis block (earliestBlock=0) is a legitimate counter value. Uninitialised state is signalled by UpdatedAt <= 0, never by Value == 0. Source: data/connector.go:L32-34
2. Stale lock expiry is silent. If a remote operation takes longer than lockTtl, the lock key expires. Unlock returns "lock was already expired" logged at DEBUG — expected and benign. Setting lockTtl < fallbackTimeout is rejected by validation to prevent this in every call. Source: common/validation.go:L292-295
3. Remote-ahead adoption happens in background. A fresh pod can have its local counter bumped forward by a peer before the pod's first RPC poll completes. Source: data/shared_state_variable.go:L686-691
4. Small-rollback rejection advances local timestamp. When a remote update has a newer UpdatedAt but a value within the rollback threshold, advanceTimestampPast is called to ensure the local (higher) value wins the next reconciliation. Source: data/shared_state_variable.go:L211-218
5. TryUpdate must not acquire updateMu. Holding updateMu during TryUpdate would block a concurrent TryUpdateIfStale caller already holding the mutex that could in turn wait for background operations — a classic deadlock. The old design had this bug; test TestLockOrderSimulation_OldBuggyBehavior documents it. Source: data/shared_state_variable_deadlock_test.go:L95-212
6. continueAsyncRefresh goroutine leak protection. If the downstream RPC ignores its context and never returns, a bounded timer of fallbackTimeout + 1s kills the helper. Source: data/shared_state_variable.go:L511
7. Double-checked staleness. TryUpdateIfStale checks IsStale before and after acquiring updateMu to prevent a thundering herd where N goroutines all see stale and all fire a refresh after the first one already updated the value. Source: data/shared_state_variable.go:L399-413
8. Redis pubsub channel buffered=1, drop-on-full. If the registry consumer falls behind, messages are silently dropped. The 5-minute periodic pollingLoop is the recovery path. Source: data/redis_pubsub_manager.go:L450-457
9. DynamoDB PublishCounterInt64 is a no-op. Cross-pod latency is bounded by statePollInterval (default 5 s). Using DynamoDB means significantly higher convergence latency than Redis or PostgreSQL. Source: data/dynamodb.go:L825-829
10. WatchCounterInt64 channel closed on shutdown. RedisPubSubManager.stop() closes all subscriber channels on app shutdown; the registry goroutine observing the channel receives ok=false and marks the watch task as failed. This is benign since the app context is also cancelled at that point. Source: data/shared_state_registry.go:L200-207
11. clusterKey propagation order. Explicit database.sharedState.clusterKey wins over the inherited top-level clusterKey. Source: common/defaults.go:L803-805
12. The served tip is per instance. Pods compute it from the same shared per-upstream heads; nothing about the pick itself is stored. Source: erpc/networks.go:L55-78
13. init.go treats shared-state failure as a warning, not fatal. A broken Redis config produces degraded per-pod behavior, not a hard startup failure. Source: erpc/init.go:L93-97
14. database.sharedState nil means SetDefaults is never called on it. The synthesised config at erpc/erpc.go:L49-L66 exists only in local scope and does not write back to cfg.Database.SharedState. Source: common/defaults.go:L837-840
15. Connector id auto-assignment differs between shared-state and other scopes. Shared-state connector id defaults to string(driver) (e.g. "redis"), not "shared-state-redis". Source: common/defaults.go:L799-801

Observability

OTel trace spans:

• CounterInt64.TryUpdate — every foreground local-update call; attributes: key, new_value (when detailed tracing enabled).
• CounterInt64.TryUpdateIfStale — attributes: key, staleness_ms, skipped_not_stale, skipped_not_stale_after_mutex, role, result_path (got_result vs timeout), async_refresh, foreground_remote_io_disabled.
• CounterInt64.TryUpdateIfStale.AcquireMutex — measures mutex wait time inside TryUpdateIfStale.
• CounterInt64.TryUpdateIfStale.ExecuteRefresh — spans the actual RPC call inside the refresh goroutine; attribute timeout_ms.
• RedisConnector.PublishCounterInt64 — spans each publish call; attributes key, value, updated_at, updated_by (when detailed tracing enabled).
• PostgreSQLConnector.PublishCounterInt64 — same pattern as Redis span above.

Key log messages (component sharedState):

• "no remote initial value found for counter" (DEBUG) — normal first-run; counter seeds from zero.
• "fetched initial value for counter" (DEBUG) — successful seed from remote on first bootstrap.
• "received new value from shared state sync" (DEBUG) — pubsub/watch update received and processed.
• "failed to setup counter sync" (ERROR) — watch setup failed; will retry via initializer.
• "lock held by another instance, proceeding with local lock" (DEBUG) — normal under load; background push proceeds without distributed lock.
• "lock acquisition timed out waiting for other instance" (WARN) — lock contention above lockMaxWait; falling back to publish-only propagation.
• "lock expired during operations (expected behavior)" (DEBUG) — expected when operations take near lockTtl.
• "published counter value to remote" (DEBUG) — successful remote publish.
• "pubsub reconnected successfully" (INFO) — Redis pubsub recovery.
• "counter value increased (remote)" (TRACE) — value advanced by remote pubsub event.
• "small rollback ignored (remote)" (TRACE) — rollback inside threshold, discarded.
• "large rollback applied (remote)" (TRACE) — real reorg or provider reset accepted.
• "large rollback held until the lower head moves (remote)" / (local) (DEBUG) — a drop beyond the threshold waits for a second, higher reading.

Source code entry points

• data/shared_state_registry.go:L38-L80 (opens in a new tab) — NewSharedStateRegistry: registry construction, clusterKey prefix, sync.Map of live counters
• data/shared_state_variable.go:L364-L707 (opens in a new tab) — TryUpdate, TryUpdateIfStale, scheduleBackgroundPushCurrent, lock ordering, rollback handling, timestamp CAS helpers
• data/shared_state_registry.go:L102-L222 (opens in a new tab) — GetCounterInt64: idempotent counter creation, bootstrap tasks, WatchCounterInt64 goroutine
• data/redis_pubsub_manager.go:L155-L460 (opens in a new tab) — RedisPubSubManager: single PSubscribe("counter:*"), reconnect backoff, periodic poll fallback, copy-on-write subscriber slice
• erpc/erpc.go:L49-L66 (opens in a new tab) — in-memory fallback synthesis when sharedState == nil
• common/defaults.go:L789-L828 (opens in a new tab) — SharedStateConfig.SetDefaults; all default values including connector id override
• data/shared_state_sources.go (opens in a new tab) — named constants for the source field in logs and traces (UpdateSourceRemoteSync, UpdateSourceTryUpdate, etc.)
• data/shared_state_variable_deadlock_test.go:L95-L212 (opens in a new tab) — lock-ordering correctness tests

Related pages

• Connectors — the backing store options (Redis, PostgreSQL, DynamoDB, memory) that power shared state.
• Tag-aware routing — use-upstream selectors scope the served tip to the matched upstreams; no shared counter is involved.
• EVM state poller — the consumer that calls TryUpdateIfStale per upstream; registers OnValue/OnLargeRollback callbacks.
• Survive provider outages — routing quality under failover depends on consistent block-height awareness across pods.
• Deployment — horizontal scaling guidance where shared state becomes necessary.