Visitar URL original
Merge main into v4 (3.4.31): MOVING drain hold, CLUSTER SLOT-STATS, keyed routing tests, SER1xx experiment block by mgravell · Pull Request #3355 · seredis/StackExchange.Redis · GitHub
Skip to content

Merge main into v4 (3.4.31): MOVING drain hold, CLUSTER SLOT-STATS, keyed routing tests, SER1xx experiment block - #3355

Merged
mgravell merged 15 commits into
v4from
marc/v4-main-sync
Nov 13, 2026
Merged

mgravell merged 15 commits into
v4from
marc/v4-main-sync

Conversation

@mgravell

Copy link
Copy Markdown
Member

Merges the fourteen commits main gained since the 3.4.29 sync. Where main's change targets code this branch deleted (PhysicalBridge, PhysicalConnection.Read, Message/ResultProcessor), it is ported by intent to the context core; where the 3.4.29 intent merge already carried the content, the conflict resolves to this side.

Ported by intent

Found by the merge

  • Heartbeat consistency pings doubled: main's assertion fix (Check the maximum operation count in HeartbeatConsistencyCheckPingsAsync #3342) turned a tautological bound into a real one, which exposed that this core sent the every-heartbeat consistency ping on the subscription connection too - the v3 bridge gates it on ConnectionType.Interactive. Now matched; the subscription connection keeps the idle-based keep-alive.
  • Experiment ID collisions: 3.x ships SER0xx numbers as features release, so this branch can never safely claim the next free one (SER010 before, now SER011/SER012). Branch-only experiments move to a SER1xx block: HandWrittenRespFragment SER011→SER101, BorrowedValues SER012→SER102, Batching SER014→SER103, with the convention recorded in Experiments.cs, docs/exp pages renamed, and NoWarn updated (SER101 deliberately absent, as SER011 was).

Resolved to this side after verifying content

Lazy subscription connection, batch profiling, IPv6 parse and the API promotion were already here from the 3.4.29 intent merge. Also taken: seredis org links (including two stragglers in comments), LICENSE split, RedisType.Bitmap/keyspace tokens, and main's coverage-ledger drift folded into this branch's adapted audit skill (SLOT-STATS row resolved).

Tests

Full repo build green (analyzers + PublicAPI). net10 suite: 7703 total, 9 failures before the heartbeat fix - 8 are the known environmental ZSCAN-duplicate issue on the dev server (fixed in the server's 8.12 branch), the 9th was the heartbeat double-ping above, now green 3/3 along with the maintenance, config and cache suites.

mgravell and others added 15 commits October 30, 2026 10:15
The dedicated pub/sub connection (always under RESP2; under RESP3 unless
SharedSubscriptionConnection is enabled) was opened eagerly as part of
activating every endpoint, so a client that never subscribes to anything
still paid for a second socket per node, and a second reconnect per node
on every failover. Seen on AMR, whose options provider disables the
configuration channel: 3.4.x showed two RESP3 connections where 3.0.x
showed one.

ActivateServer now opens that connection only when there is something to
subscribe to straight away: a configuration channel, or subscriptions that
already exist (and may need to move to this endpoint). Otherwise it is
created on first use, via the existing GetBridge paths; with the default
backlog policy the first SUBSCRIBE queues until it is up.

The connect wait ("second leg") and the maintenance handoff deadline
treated an absent subscription bridge as a disconnected one, which would
have left connects hanging; they now consider a connection that has not
been asked for as not required. The endpoint-state connect log says
"NotRequired" for that case instead of "Disconnected".

Tests: AzureManagedRedisConnectionCount now asserts one connection before
pub/sub and the expected count after; a new test covers the eager case
when a configuration channel is set; two tests that relied on the eager
subscription connection now subscribe first.
* Command coverage ledger and audit skill

Adds docs/CommandCoverage.md: every upstream Redis command the library does
not expose as a typed API, grouped and with a recorded decision (policy,
not applicable, deferred, todo, deprecated, enum-only, implemented). Seeded
from a diff of redis/redis src/commands (477 specs, @6a5fee67) against the
RedisCommand enum and source; it carries forward the dispositions from
#2055 (blocking is policy, Functions and ACL are deferred) and replaces
that hand-maintained list.

Adds the audit-command-coverage skill and its script, which re-runs the
diff on request and reports only what changed: new upstream commands with
no decision, ledger rows the code now covers, rows the spec no longer has,
and enum members the spec does not know. The script exits non-zero when
the ledger needs editing.

Closes #2055.

* Ledger: CLIENT TRACKING is v4, cluster/COMMAND introspection is administration, CLUSTER SHARDS note corrected

* Ledger: CLIENT PAUSE/UNPAUSE are operator tooling, not applicable
…ent server values (#3312)

* Object-type metadata for native bitmaps (redis/redis#15331), plus the tokens that diff turned up

RedisType gains Bitmap ("bitmap", the TYPE reply for a native roaring
bitmap, 8.12) and Gcra ("gcra", already in the server's type-name table).
KeyNotificationType gains SetBit (emitted under class $ for string bitmaps
and class b for native ones) and the other event names the server emits
that the enum lacked: arinsert, armset, arring, arseek, gcra, pfadd,
sexpire, spersist, sinterstore, zrangestore, and keymiss (class m, side
effect group).

Fixes the Copy token: the server has only ever emitted "copy_to", so the
"copy" token never parsed and a RedisChannel.KeyEvent(KeyNotificationType.Copy)
subscription never fired. The member and its value are unchanged.

Both parsers are generated from [AsciiHash]; tests extend the token tables.

* Drop Gcra: the type was withdrawn before release, so there is nothing for the token to match

* Add sunionstore/sdiffstore events (from the keyspace-notifications docs); drop the history comment on Copy
* Forward the name argument in ConnectionGroupMember's string constructor

The string overload chained to the ConfigurationOptions constructor without passing `name`, so every
member built from a configuration string (the idiom used throughout docs/Failover.md) was named after
its first endpoint by Init() instead of the name the caller gave it. Regression test added.

* Parenthesize a Debug-only assertion that trips SA1407

`endSegmentLength == 0 & endSegmentCommitted == 0` in DebugAssertValid sits inside `#if DEBUG`, so CI
(Release) never compiles it and never saw the StyleCop error; any local Debug build of RESPite failed
with TreatWarningsAsErrors. Introduced by ee47e53 after the StyleCop bump in aa1e5c0.
* Parse unbracketed IPv6 ip:port from the server (#1422)

CLUSTER NODES, MOVED/ASK and CLIENT LIST write IPv6 addresses without
brackets, i.e. "2a06:4e80:3:a::a:7000". The RFC 3986 parser used for
configuration strings reads that as a (valid!) eight-group address with
no port, so cluster nodes ended up as "[2a06:4e80:3:a::a:7000]:0" and
redirections and client addresses were equally wrong.

Add Format.TryParseServerEndPoint for the server's "ip:port" form: when
there is no ']' and more than one ':', split at the last colon and
require both halves to parse; anything else goes to the existing parser
unchanged, so configuration parsing is not touched. Use it at the three
sites that receive server-emitted text.

The sentinel +switch-master handler gets host and port as separate
tokens and was gluing them back together with ':' before parsing; pass
them to the (host, port) overload instead.

* Parse unbracketed IPv6 in MOVING/SMIGRATED endpoints

ParseMigrationEndPoint used the configuration-string parser, so an unbracketed IPv6 address parsed as an address with port 0, which the placeholder check then discarded. Use TryParseServerEndPoint, as the other server-emitted sites do; tests for the MOVING successor and SMIGRATED source/target.
Co-authored-by: mohsinaliwork686-rgb <229973101+mohsinaliwork686-rgb@users.noreply.github.com>
…since 3.4.0 (#3316)

* public-api.py: apply *REMOVED* entries at promotion instead of copying them

The analyzer records a vanished shipped signature as a *REMOVED* line in
the unshipped file. Promotion moved every unshipped line across verbatim,
so a promotion after any such change would have left the shipped file
with both the dead signature and a *REMOVED* copy of it, and the analyzer
would then fail the build on the dead one (RS0017).

Promotion now consumes the marker: the named signature is dropped from the
shipped file, the marker is not carried over, and a marker that names
nothing shipped is an error. A *REMOVED* line already sitting in a shipped
file is also reported. The "+N promoted" note gains "-N removed", and an
unshipped file holding only removals is still emptied.

* Promote the shipped API surface: everything since 3.4.0

eng/public-api.py --promote. Fifty-two StackExchange.Redis entries (the
lag-aware availability surface, FailbackHealthCheck, HealthCheckProbeScope,
the 8.12-independent RedisType and KeyNotificationType tokens) and the
net6.0 SetUserPemCertificate overload move to shipped; the three parameter
renames from #3296 (muxer -> multiplexer on the Azure providers'
AfterConnectAsync, other -> obj on RESPite AsciiHash.Equals) are applied
rather than copied. Every unshipped file is now empty.
* Split third-party notices out of LICENSE so GitHub detects MIT

The LICENSE file carried ~1.1KB of third-party notes after the MIT text,
which drops it below licensee's similarity threshold; GitHub reports the
repo license as "Other" (SPDX NOASSERTION). Keep LICENSE as the bare MIT
text and move the notes, verbatim, to THIRD-PARTY-NOTICES.md.

* Correct the third-party notices

- Redis: BSD-3 only through 7.2; RSALv2/SSPLv1 from 7.4, AGPLv3 added in
  8.0; point at redis.io/legal/licenses (the old /topics/license URL now
  redirects to the docs landing page).
- redis-doc: repo is archived under redis/redis-doc; note that redis/docs,
  its successor, is CC BY-NC-SA for newer content.
- Drop the Redis-64 and BookSleeve notes: neither is part of the
  development solution any more.
- Add the code actually incorporated from other projects (dotnet/runtime,
  dotnet/corefx, aspnet/BasicMiddleware) and the test-only redis-sharp copy,
  none of which the old notes mentioned.
…queue; a close with commands in flight no longer completes a call with the previous call's reply (#3330)

RecordConnectionFailed drains _writtenAwaitingResponse one message at a
time, completing each outside the lock, while the read loop can still be
matching replies that were already in the receive buffer. Each reply the
loop matched after the drain had taken a message went to the message
*after* the one it answered: GET key N completed, successfully, with key
N-1's value, and every reply behind it was off by one. Reproduced on
main with the reporter's loop: 616 wrong replies in 20,000 calls.

_isShutdown was already set under the queue's lock before the drain, and
the loop already consulted it - but only once the queue was empty.
Checking it before the dequeue closes the race: a reply arriving after
teardown has begun is dropped with the connection, and the message it
answered is failed by the drain as before.

The test re-creates the scenario against the real server (20 rounds of
500 in-flight GETs closed with allowCommandsToComplete: false) and
asserts that no call completes with another call's value; without the
fix it sees ~200 in a run.

Fixes #3329
…imental SER012 (#3341)

* CLUSTER SLOT-STATS (8.2): per-slot usage statistics on IServer, experimental SER012

IServer.ClusterSlotStatsBySlotRange(Async) and ClusterSlotStatsOrderBy(Async), modelled
on the CLUSTER SLOTS file layout and the HOTKEYS field handling: node-local read, lenient
nominal parsing, config-gated metrics as nullable properties (absent != zero), unknown
metrics preserved as Metadata, cpu-usec surfaced as TimeSpan. RedisFeatures.ClusterSlotStats
gates on 8.2 RC1. Promotes the CommandCoverage 'deferred' row.

* The storage-kind tripwire acknowledges SLOT-STATS and SLOTSRANGE: two new >8-byte literals, 28 -> 30
…kExchange/StackExchange.Redis -> github.com/seredis/StackExchange.Redis everywhere, including the nupkg RepositoryUrl/PackageReleaseNotes metadata (#3344)
…ands drain, so the recycle happens within a round trip instead of at the window end (#3347)

* MOVING handoff: hold new work in the backlog while the in-flight commands drain, so the recycle happens within a round trip instead of at the window end

The drain waited for "no caller work", which under steady traffic never comes: there is always a command in flight, so the recycle waited out the whole announced window and failed whatever was in flight at that instant. Measured live (endpoint rebind, single proxy, ~900 ops/s): recycle at +20.0s against a MOVING at +6.0s, 16 failed commands, missed-deadline warning.

While a handoff drains, the bridge holds a caller's work in the backlog instead of writing it to the connection about to be replaced; the backlog processor does not flush while the hold stands; the drain waits only for what is already written, capped at half the relaxed timeout (bounded by half the remaining window, floor one second); the connection is then recycled as before, the hold cleared, and the held work flushed to the replacement by the ordinary reconnect path. Timeouts are relaxed for the hold and the reconnect. A FailFast policy does not abandon held work for the handoff's own disconnect. Same scenario on this branch: recycle at +6.2s against a MOVING at +5.9s, 0 failed of 97k.

Deliberately a flag and nothing more: no per-message timeout cover (with relaxation off and a timeout shorter than the hold, held commands can time out in the backlog, as on any reconnect), and no retire under the write lock (a writer that reaches the lock after the recycle gets a write fault, as before). Also: the drain budget is recomputed after the DNS poll, a second MOVING during a handoff updates the successor pin rather than being dropped, and the pin lasts the window so the second bridge dials the same successor.

Tests: steady load (with and without relaxation), slow command inside the cap, blocking command past the cap; plus a fault-injector tier measurement that records the live timeline.

* OneLogicalEventFromEveryNodeRaisesOneEvent: wait for the one event rather than counting at the instant the second node relaxed; the callback is dispatched after the relaxation, so the count could be read as zero (flaked on main at ~1 in 6)
IncrexMessageBase (#3065) and ArrayGrepMessage (#3076) derived from Message
directly, held a key, and never overrode GetHashSlot, so they reported
NoSlot. On a cluster that sends the command to an arbitrary node and leans
on -MOVED: an extra round trip for most calls, and a RedisServerException
under CommandFlags.NoRedirect.

Both now derive from Message.CommandKeyBase, as their single-key siblings
do, which supplies the slot (and the key in CommandAndKey).

Affects the StringIncrement overloads that take an Expiration (INCREX) and
ArrayGrep. No public API change.
…eyed routing tests, SER1xx experiment block

Fourteen commits of drift, merged by intent where main's code no longer exists:

* MOVING handoff (#3347): ported to the context core. RespEndpointExecutor
  gains a handoff hold - new caller work goes to the backlog beside the
  write-slot mechanism, internal traffic bypasses it, FailFast does not
  abandon held work, and the backlog does not flush while it stands. The
  drain waits only for written caller work, capped at half the relaxed
  timeout (never more than half the remaining window, floor one second),
  the budget is recomputed after the DNS poll, timeouts are relaxed for
  the hold, and a second MOVING updates the successor pin.
* Read-loop teardown (#3330): already closed in RESPite by design - the
  _pendingLock serialises reply-matching against Close, and its comment
  cites the same #3329; the public regression test merges as-is.
* INCREX/ARGREP slot routing (#3352): structurally subsumed - the builder's
  key append folds the slot - so the fix arrives as tests: FakeExecutor now
  captures per-request slots, plus the cluster NoRedirect integration pair.
* CLUSTER SLOT-STATS (#3327): public surface as main shipped it; the
  composition is two IRespArgument operands on RedisServer (the HotKeys
  pattern) over new RespLiterals fragments, parsing is an IRespHandler port
  of the processor, NodeLocalRead retry category, ledger row resolved.
* Heartbeat consistency pings are interactive-only, as the v3 bridge gated
  them; main's assertion fix (#3342) exposed that this core pinged the
  subscription connection every heartbeat too, doubling the operation count.
* Experiment IDs: the 3.x line ships SER0xx numbers, so branch-only
  experiments move to a SER1xx block it can never race - SER011->SER101
  (hand-written fragments), SER012->SER102 (borrowed values, as 3.x released
  SER012 for ClusterSlotStats), SER014->SER103 (batching) - with the
  convention recorded in Experiments.cs and the docs/exp pages renamed.
* Already here by content from the 3.4.29 intent merge, resolved to this
  side: lazy subscription connection, batch profiling, IPv6 parse, API
  promotion. Also taken: seredis org links, LICENSE split, the
  RedisType.Bitmap/keyspace tokens, and main's coverage-ledger drift folded
  into this branch's adapted audit skill.
@mgravell
mgravell merged commit c76d4e6 into v4 Nov 13, 2026
3 of 5 checks passed
@mgravell
mgravell deleted the marc/v4-main-sync branch November 13, 2026 13:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants