Visitar URL original
Redis native compressed bitmaps (redis-roaring) by aviggiano · Pull Request #15331 · redis/redis · GitHub
Skip to content

Redis native compressed bitmaps (redis-roaring) - #15331

Merged
sundb merged 749 commits into
redis:unstablefrom
aviggiano:unstable
Oct 14, 2026
Merged

sundb merged 749 commits into
redis:unstablefrom
aviggiano:unstable

Conversation

@aviggiano

@aviggiano aviggiano commented Jun 11, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #15296

Native Roaring bitmaps

Adds a native Roaring bitmap representation for Redis bitmaps, backed by vendored CRoaring v4.7.2 (deps/croaring, built with SIMD disabled and symbols hidden for portability). A string bitmap costs one byte per 8 bit positions up to its highest offset, regardless of how many bits are set; sparse or large bitmaps are instead stored compressed, cutting memory dramatically while preserving exact bit semantics.

Type & object model

  • New object type OBJ_BITMAP with encoding OBJ_ENCODING_BITMAP_ROARING. TYPE → bitmap, OBJECT ENCODING → bitmap-roaring.
  • Object header: encoding widened 4→5 bits to fit the new encoding value; to keep redisObject the same width, refcount drops 23→22 bits, lowering the maximum non-shared refcount from 8388605 to 4194301 (still far above any real per-object refcount).

Enabling it — bitmap-default-roaring

  • New boolean config (default no, runtime-modifiable). When off, bitmap writes keep creating plain strings. Native bitmaps remain supported and can enter the keyspace through BITOP with a Roaring source, RESTORE, replication, or loading an RDB/AOF containing them.
  • When on, bit-writes that create or extend a bitmap produce a native Roaring bitmap, an existing string bitmap converts in place on the next bit-write (TTL and module metadata preserved), and BITOP destinations are Roaring. Automatic creation/conversion is disabled on replicas and when a write's propagation excludes either AOF or replicas — notably RM_Call() without !, selective RM_Call() flags, and scripts using redis.set_repl(); such writes keep plain strings on every node.

Command surface & string boundary

  • Bit commands operate on both representations transparently: SETBIT, GETBIT, BITCOUNT, BITPOS, BITOP, BITFIELD, BITFIELD_RO.
  • A native Roaring bitmap is a distinct type: plain string commands (GET, GETRANGE, APPEND, SETRANGE, STRLEN, …) return WRONGTYPE. DEBUG BITMAP-RAW exposes the materialized bytes.
  • Bit offsets follow the historical string rule on both representations: capped at proto-max-bulk-len × 8. Internal representability is bounded at 2^60−1 logical bytes (BITROAR_MAX_BYTES). BITOP NOT over a Roaring source bounds missing-chunk allocation with a budget derived from proto-max-bulk-len (minimum 65,536 chunks of 8 KiB). Replication and AOF replay bypass this budget to apply accepted operations; dense string sources are exempt.

Persistence

  • Adds RDB_TYPE_BITMAP (33) under the existing RDB version 16 introduced for 8.12 (this PR does not bump it). This relies on shipping in 8.12: 8.12 milestone builds without this PR reject type 33 cleanly, and if the PR slips past 8.12, RDB_VERSION must be bumped. The RDB/DUMP payload is the logical byte length followed by the CRoaring64 portable serialization — compact, not a dense byte expansion. Loading enforces the representability bound and validates the blob.

Replication & AOF — deterministic by construction

  • Representation decisions are never re-derived from a replica's or AOF-loader's own config. The transition travels in the stream as the internal BITCONVERT <key> and BITROAROP commands (hidden from COMMAND, internal-client gated, since 8.12.0; ACL CAT and ACL DRYRUN are not yet filtered and still list them):
    • SETBIT/BITFIELD creating or converting a bitmap replay as BITCONVERT followed by the original command, grouped in MULTI/EXEC;
    • a BITOP whose destination is Roaring only by configuration replays as internal BITROAROP, which stores the native result directly and preserves notification and callback ordering;
    • operations on already-Roaring operands propagate verbatim.
  • BITCONVERT applies the same in-place transition the primary performed: TTL and module metadata retained, no value payload in the stream.

Notifications, modules, cluster

  • NOTIFY_BITMAP (class b) keyspace events; conversion is an observable in-place type change: type_changed (class c) then the command's own event when it changes the logical value — never overwritten. A conversion that leaves the logical value unchanged still emits type_changed, counts as a write, and is propagated. A newly created native key emits new (class n) rather than type_changed.
  • Modules observe the transition via the type-changed callback; REDISMODULE_KEYTYPE_BITMAP added.
  • Works under Cluster, including atomic slot migration (tests/unit/cluster/bitmap.tcl, atomic-slot-migration.tcl coverage).

Memory accounting & active defrag

  • Full memory accounting; all CRoaring allocations route through zmalloc. Active defrag walks the Roaring64 ART and its ARRAY, BITSET, and RUN containers (via a small vendor patch exposing roaring64 internals), and fork-child memory is dismissed correctly. INFO keysizes gains a distrib_bitmaps_sizes histogram of logical byte length, not memory: a sparse Roaring bitmap can be orders of magnitude smaller than its bucket, unlike distrib_<collection>_sizes, which reports allocation size. Bitmap allocation sizes are tracked internally but, as for strings, not reported by INFO.

Testing

  • tests/unit/bitroar.tcl (encoding, conversion, RDB/AOF/replication determinism incl. mode-mismatch masters/replicas, defrag, module boundary), string↔Roaring parity oracle cases in the same suite, plus additions to bitops.tcl, memefficiency.tcl, corrupt-dump-fuzzer.tcl, info-keysizes.tcl, integration/replication-3.tcl, integration/dismiss-mem.tcl, cluster suites, moduleapi/keyspace_events transition-callback replay coverage, and moduleapi/propagate selective-propagation coverage.

Documentation


Note

Medium Risk
Adds a large vendored dependency with local patches to internal 64-bit layout; incorrect integration could affect build portability and future memory/defrag paths that walk CRoaring allocations.

Overview
Vendors CRoaring v4.7.2 under deps/croaring as the C Roaring bitmap engine for Redis’s native compressed bitmap encodings, with upstream sources plus Redis-maintained build/docs (not upstream CMake).

Build integration: deps/Makefile gains a croaring target that produces libroaring.a with hidden symbol visibility, x64 SIMD and NEON disabled (ROARING_DISABLE_X64, DISABLENEON), and LTO not inherited from Redis so CI/cross-builds stay linkable. distclean and .gitignore cover the new archive; AR is AR?= so cross-compile toolchains can override the archiver.

Redis-specific maintenance: deps/README.md documents upgrade steps, local patches (portability/endian/atomics in portability.h, and 64-bit internal layout exposure for bitroar memory accounting, fork-child dismissal, and active defrag), and a small refcount-none regression test in the croaring Makefile.

Reviewed by Cursor Bugbot for commit 8963e1c. Bugbot is set up for automated code reviews on this repo. Configure here.

@jit-ci

This comment was marked as resolved.

Comment thread src/defrag.c Resolved
Comment thread src/rdb.h Resolved
Comment thread src/object.h Resolved
Comment thread src/lazyfree.c Resolved
Comment thread src/bitops.c Resolved
Comment thread src/bitops.c Resolved
@sundb

This comment was marked as resolved.

Comment thread src/bitmap_roaring.c Resolved
Comment thread src/rdb.h Resolved
Comment thread redis.conf Resolved
Comment thread src/rdb.c Resolved
Comment thread src/bitmap_roaring.c Resolved
Comment thread src/bitmap_roaring.c Resolved
Comment thread src/bitops.c Resolved
Comment thread src/bitmap_roaring.c Resolved
Comment thread src/bitmap_roaring.c Resolved
Comment thread src/debug.c Resolved
Comment thread src/bitmap_roaring.c Resolved
Comment thread src/debug.c Resolved
Comment thread src/debug.c Resolved
Comment thread src/bitops.c Resolved
Comment thread src/bitops.c Resolved
Comment thread src/bitops.c Resolved
Comment thread src/bitops.c Resolved
Comment thread src/bitops.c Resolved
Comment thread src/rdb.c Resolved
Comment thread src/server.h Resolved
Comment thread src/rdb.c Resolved
Comment thread src/rdb.c Resolved
Comment thread src/bitops.c Resolved
@sundb
sundb self-requested a review June 22, 2026 05:03
aviggiano added a commit to aviggiano/redis that referenced this pull request Oct 13, 2026
aviggiano added a commit to aviggiano/redis that referenced this pull request Oct 13, 2026
aviggiano added a commit to aviggiano/redis that referenced this pull request Oct 13, 2026
aviggiano added a commit to aviggiano/redis that referenced this pull request Oct 13, 2026
aviggiano added a commit to aviggiano/redis that referenced this pull request Oct 13, 2026
aviggiano added a commit to aviggiano/redis that referenced this pull request Oct 13, 2026
aviggiano added a commit to aviggiano/redis that referenced this pull request Oct 13, 2026
aviggiano added a commit to aviggiano/redis that referenced this pull request Oct 13, 2026
aviggiano added a commit to aviggiano/redis that referenced this pull request Oct 13, 2026
The reorder guarded the keysizes/slot accounting against a module
"new"-event callback synchronously deleting the key and leaving 'kv'
dangling. Review of redis#15331 confirmed this notification
ordering issue is a pre-existing upstream bug with an internal fix
(RED-200931) that will be backported, so this PR should not carry its
own fix. Restore the exact upstream ordering; dbAddInternal() now
matches redis/redis unstable and the hunk disappears from the upstream
diff.

Nothing on the bitmap side depends on the ordering: *valref is
assigned before dbAddInternal() returns either way, and the
setKeyByLink() subscriber gate operates after the call returns.

Fixes #165
aviggiano added a commit to aviggiano/redis that referenced this pull request Oct 13, 2026
Addresses every unresolved review thread from sundb on redis#15331
as of July 9:

- bitops.c: revert the SETBIT/BITFIELD keysizes-histogram blocks to the
  upstream condition and statement order, dropping the created out-param
  from lookupStringForBitCommand(). The pre-existing-empty-string
  histogram bug is tracked separately in #168 for its own upstream PR.
- bitops.c: extract bitmapResolveNativeTarget(), a single helper that
  decides whether a SETBIT/BITFIELD write targets a native bitmap and
  performs the create-or-convert transition; both commands now share it.
  The standalone bitmapNativeOffsetWithinLimit() helper is deleted and
  its native-cap preflight lives inside the shared helper (still needed
  when proto-max-bulk-len is raised above the fixed 512MB native cap).
- bitops.c: fold the signed bitfield sign-extension into
  getSignedBitfieldFromValue() via getUnsignedBitfieldFromValue() and
  delete getSignedBitfieldFromBitmapObject().
- bitops.c: document bitopUseMixedRawBytePath()'s size/density
  heuristic.
- rdb.c: load native bitmap payloads with rdbGenericLoadStringObject()
  plus a post-load length check, deleting the near-duplicate
  rdbLoadBitmapRawString()/rdbLoadBitmapLzfRawString() loaders.
- db.c: revert setKeyByLink() to the upstream keyModified() call.
- bitmap_roaring.h: expose only BITMAP_OBJECT_MAX_BYTES; the bit-offset
  form moves into bitmap_roaring.c.
- dismiss-mem.tcl: the bitmap dismiss test now covers all three roaring
  container types (bitset, run, array).

Fixes #167
aviggiano added a commit to aviggiano/redis that referenced this pull request Oct 13, 2026
The stream.tcl change is a standalone flaky-test fix unrelated to the
bitmap->roaring conversion; it now lives in redis#15507. Restore
stream.tcl to match upstream/unstable so PR redis#15331 contains only the
bitmap changes.
@aviggiano

Copy link
Copy Markdown
Contributor Author

@sundb Could you please re-review the latest changes? The remaining October 12 feedback is addressed, the upstream conflict is resolved, and the CLA check now passes. The feedback changes remain in separate commits.

@sundb

sundb commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator

@sundb
sundb merged commit f3b9b68 into redis:unstable Oct 14, 2026
19 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in Redis 8.12 Oct 14, 2026
paulorsousa added a commit to paulorsousa/redis that referenced this pull request Oct 14, 2026
redis#15331 made bitopCommandGeneric() look every source up twice (once for the
lookup side effects, once to collect the values) and look the target up
before reading the sources. With 10 sources that is 21 lookups instead of 10.

Collect each source in the same pass that looks it up, as before redis#15331.
A module callback that writes to a source that was already looked up, while
a later source is being looked up, is not guarded against anymore: writes
from keyspace notification callbacks are documented as dangerous and
discouraged (see RM_SubscribeToKeyspaceEvents, RM_AddPostNotificationJob).

The target is still looked up right before a native result is stored, so a
logically expired target is expired first and its DEL is queued ahead of the
BITOP. Otherwise replicas and the AOF would delete the result.

Remove the two keyspace_events tests that asserted the previous two-pass
contract.
sundb pushed a commit that referenced this pull request Oct 16, 2026
…nce (#15970)

## Summary

#15331 made `bitopCommandGeneric()` look every source up twice (once for
the lookup side effects, once to collect the values) and look the target
up before reading the sources. For 10 sources that is 21 lookups per
call instead of 10, and for `BITOP` over many short strings the lookups
dominate the work.

This collects each source in the same pass that looks it up, as before
#15331: one `lookupKeyRead()` per source.

Two behaviours follow from that:

- **Module callbacks that write.** Writing from a keyspace notification
callback is documented as dangerous and highly discouraged
(`RM_SubscribeToKeyspaceEvents`, `RM_AddPostNotificationJob`). `BITOP`
no longer guards against it: if a callback writes to a source that was
already looked up while a later source is being looked up, the result
may not reflect that write, and a master and its replica may differ.
Roaring operands are borrowed from the keyspace and not refcounted, so a
callback that deletes or replaces one of them would leave a dangling
pointer. The set commands (`src/t_set.c`) already hold their source
pointers across later lookups in the same way.
- **The target is still looked up right before a native result is
stored.** Without it a logically expired target would expire last, so
its `DEL` would be queued after the `BITOP`, and replicas and the AOF
would delete the result. The string result path does not need it.

Thanks @sundb for the review that led to dropping the two-pass sequence.

## Behaviour differences from #15331

These match the behaviour before #15331:

- A wrong-type source together with a logically expired target: the
target is no longer expired (and no `DEL` is propagated) before the
`WRONGTYPE` error.
- An empty result into a logically expired target now emits `del`, not
`expired`, and propagates no `DEL`.

## Tests

The two `keyspace_events` tests that asserted the previous contract are
removed ("BITOP reads sources after lookup callbacks change them",
"BITOP expires its destination before reading sources"). `unit/bitops`,
`unit/bitroar`, `unit/bitfield`, `unit/expire` and
`unit/moduleapi/keyspace_events` pass. Locally the full `moduleapi`
suite aborts at an unrelated `KEYMETA - AOF rewrite` test (a Tcl read
error on this machine), so CI is the check for the rest.

Still to add: module-free tests for the native result into an expired
target (AOF and replica), a wrong-type source after decoded strings, a
target that is also a source, and duplicate sources.
mgravell added a commit to seredis/StackExchange.Redis that referenced this pull request Oct 30, 2026
…ent server values (#3312)

* Object-type metadata for native bitmaps (redis/redis#15331), plus the tokens that diff turned up

RedisType gains Bitmap ("bitmap", the TYPE reply for a native roaring
bitmap, 8.12) and Gcra ("gcra", already in the server's type-name table).
KeyNotificationType gains SetBit (emitted under class $ for string bitmaps
and class b for native ones) and the other event names the server emits
that the enum lacked: arinsert, armset, arring, arseek, gcra, pfadd,
sexpire, spersist, sinterstore, zrangestore, and keymiss (class m, side
effect group).

Fixes the Copy token: the server has only ever emitted "copy_to", so the
"copy" token never parsed and a RedisChannel.KeyEvent(KeyNotificationType.Copy)
subscription never fired. The member and its value are unchanged.

Both parsers are generated from [AsciiHash]; tests extend the token tables.

* Drop Gcra: the type was withdrawn before release, so there is nothing for the token to match

* Add sunionstore/sdiffstore events (from the keyspace-notifications docs); drop the history comment on Copy
mgravell added a commit to seredis/StackExchange.Redis that referenced this pull request Oct 31, 2026
…, batch profiling, API promotion (#3322)

* Make the dedicated subscription connection lazy (#3310)

The dedicated pub/sub connection (always under RESP2; under RESP3 unless
SharedSubscriptionConnection is enabled) was opened eagerly as part of
activating every endpoint, so a client that never subscribes to anything
still paid for a second socket per node, and a second reconnect per node
on every failover. Seen on AMR, whose options provider disables the
configuration channel: 3.4.x showed two RESP3 connections where 3.0.x
showed one.

ActivateServer now opens that connection only when there is something to
subscribe to straight away: a configuration channel, or subscriptions that
already exist (and may need to move to this endpoint). Otherwise it is
created on first use, via the existing GetBridge paths; with the default
backlog policy the first SUBSCRIBE queues until it is up.

The connect wait ("second leg") and the maintenance handoff deadline
treated an absent subscription bridge as a disconnected one, which would
have left connects hanging; they now consider a connection that has not
been asked for as not required. The endpoint-state connect log says
"NotRequired" for that case instead of "Disconnected".

Tests: AzureManagedRedisConnectionCount now asserts one connection before
pub/sub and the expected count after; a new test covers the eager case
when a configuration channel is set; two tests that relied on the eager
subscription connection now subscribe first.

* Command coverage ledger and audit skill (#3311)

* Command coverage ledger and audit skill

Adds docs/CommandCoverage.md: every upstream Redis command the library does
not expose as a typed API, grouped and with a recorded decision (policy,
not applicable, deferred, todo, deprecated, enum-only, implemented). Seeded
from a diff of redis/redis src/commands (477 specs, @6a5fee67) against the
RedisCommand enum and source; it carries forward the dispositions from
#2055 (blocking is policy, Functions and ACL are deferred) and replaces
that hand-maintained list.

Adds the audit-command-coverage skill and its script, which re-runs the
diff on request and reports only what changed: new upstream commands with
no decision, ledger rows the code now covers, rows the spec no longer has,
and enum members the spec does not know. The script exits non-zero when
the ledger needs editing.

Closes #2055.

* Ledger: CLIENT TRACKING is v4, cluster/COMMAND introspection is administration, CLUSTER SHARDS note corrected

* Ledger: CLIENT PAUSE/UNPAUSE are operator tooling, not applicable

* Update RedisType.Bitmap and the keyspace-event tokens to reflect current server values (#3312)

* Object-type metadata for native bitmaps (redis/redis#15331), plus the tokens that diff turned up

RedisType gains Bitmap ("bitmap", the TYPE reply for a native roaring
bitmap, 8.12) and Gcra ("gcra", already in the server's type-name table).
KeyNotificationType gains SetBit (emitted under class $ for string bitmaps
and class b for native ones) and the other event names the server emits
that the enum lacked: arinsert, armset, arring, arseek, gcra, pfadd,
sexpire, spersist, sinterstore, zrangestore, and keymiss (class m, side
effect group).

Fixes the Copy token: the server has only ever emitted "copy_to", so the
"copy" token never parsed and a RedisChannel.KeyEvent(KeyNotificationType.Copy)
subscription never fired. The member and its value are unchanged.

Both parsers are generated from [AsciiHash]; tests extend the token tables.

* Drop Gcra: the type was withdrawn before release, so there is nothing for the token to match

* Add sunionstore/sdiffstore events (from the keyspace-notifications docs); drop the history comment on Copy

* Fix ConnectionGroupMember string-ctor dropping the name (#3314)

* Forward the name argument in ConnectionGroupMember's string constructor

The string overload chained to the ConfigurationOptions constructor without passing `name`, so every
member built from a configuration string (the idiom used throughout docs/Failover.md) was named after
its first endpoint by Init() instead of the name the caller gave it. Regression test added.

* Parenthesize a Debug-only assertion that trips SA1407

`endSegmentLength == 0 & endSegmentCommitted == 0` in DebugAssertValid sits inside `#if DEBUG`, so CI
(Release) never compiles it and never saw the StyleCop error; any local Debug build of RESPite failed
with TreatWarningsAsErrors. Introduced by ee47e53 after the StyleCop bump in aa1e5c0.

* Parse unbracketed IPv6 ip:port from the server (#1422) (#3307)

* Parse unbracketed IPv6 ip:port from the server (#1422)

CLUSTER NODES, MOVED/ASK and CLIENT LIST write IPv6 addresses without
brackets, i.e. "2a06:4e80:3:a::a:7000". The RFC 3986 parser used for
configuration strings reads that as a (valid!) eight-group address with
no port, so cluster nodes ended up as "[2a06:4e80:3:a::a:7000]:0" and
redirections and client addresses were equally wrong.

Add Format.TryParseServerEndPoint for the server's "ip:port" form: when
there is no ']' and more than one ':', split at the last colon and
require both halves to parse; anything else goes to the existing parser
unchanged, so configuration parsing is not touched. Use it at the three
sites that receive server-emitted text.

The sentinel +switch-master handler gets host and port as separate
tokens and was gluing them back together with ':' before parsing; pass
them to the (host, port) overload instead.

* Parse unbracketed IPv6 in MOVING/SMIGRATED endpoints

ParseMigrationEndPoint used the configuration-string parser, so an unbracketed IPv6 address parsed as an address with port 0, which the placeholder check then discarded. Use TryParseServerEndPoint, as the other server-emitted sites do; tests for the MOVING successor and SMIGRATED source/target.

* Profile commands sent in a batch (#2175) (#3313)

Co-authored-by: mohsinaliwork686-rgb <229973101+mohsinaliwork686-rgb@users.noreply.github.com>

* public-api.py: apply *REMOVED* at promotion; promote the API surface since 3.4.0 (#3316)

* public-api.py: apply *REMOVED* entries at promotion instead of copying them

The analyzer records a vanished shipped signature as a *REMOVED* line in
the unshipped file. Promotion moved every unshipped line across verbatim,
so a promotion after any such change would have left the shipped file
with both the dead signature and a *REMOVED* copy of it, and the analyzer
would then fail the build on the dead one (RS0017).

Promotion now consumes the marker: the named signature is dropped from the
shipped file, the marker is not carried over, and a marker that names
nothing shipped is an error. A *REMOVED* line already sitting in a shipped
file is also reported. The "+N promoted" note gains "-N removed", and an
unshipped file holding only removals is still emptied.

* Promote the shipped API surface: everything since 3.4.0

eng/public-api.py --promote. Fifty-two StackExchange.Redis entries (the
lag-aware availability surface, FailbackHealthCheck, HealthCheckProbeScope,
the 8.12-independent RedisType and KeyNotificationType tokens) and the
net6.0 SetUserPemCertificate overload move to shipped; the three parameter
renames from #3296 (muxer -> multiplexer on the Azure providers'
AfterConnectAsync, other -> obj on RESPite AsciiHash.Equals) are applied
rather than copied. Every unshipped file is now empty.

* Split third-party notices out of LICENSE so GitHub detects MIT (#3317)

* Split third-party notices out of LICENSE so GitHub detects MIT

The LICENSE file carried ~1.1KB of third-party notes after the MIT text,
which drops it below licensee's similarity threshold; GitHub reports the
repo license as "Other" (SPDX NOASSERTION). Keep LICENSE as the bare MIT
text and move the notes, verbatim, to THIRD-PARTY-NOTICES.md.

* Correct the third-party notices

- Redis: BSD-3 only through 7.2; RSALv2/SSPLv1 from 7.4, AGPLv3 added in
  8.0; point at redis.io/legal/licenses (the old /topics/license URL now
  redirects to the docs landing page).
- redis-doc: repo is archived under redis/redis-doc; note that redis/docs,
  its successor, is CC BY-NC-SA for newer content.
- Drop the Redis-64 and BookSleeve notes: neither is part of the
  development solution any more.
- Add the code actually incorporated from other projects (dotnet/runtime,
  dotnet/corefx, aspnet/BasicMiddleware) and the test-only redis-sharp copy,
  none of which the old notes mentioned.

---------

Co-authored-by: Mohsin Ali <mohsin.ali.work686@gmail.com>
Co-authored-by: mohsinaliwork686-rgb <229973101+mohsinaliwork686-rgb@users.noreply.github.com>
ndyakov added a commit to redis/go-redis that referenced this pull request Nov 10, 2026
* feat(search): add FT.HYBRID RERANK option

Redis 8.8.7 added RERANK to the KNN and RANGE clauses of FT.HYBRID for
disk-based HNSW indexes. Send it inside the counted method params.

* chore(cmd): regenerate command info for 8.12-rc2

Adds set member expiration, BITRANGEINTARRAY and BLESS records and
corrects json.mget, cms.merge and ts.createrule metadata.

* test(bitmap): cover native roaring bitmaps

Pins that the bit commands, DUMP/RESTORE and SCAN TYPE work on the
bitmap type added in Redis 8.12 (redis/redis#15331).

* refactor(search): make hybrid rerank an enum

Replace the Rerank/HasRerank bool pair on FTHybridVectorExpression with
a single FTRerank value (default, enabled, disabled).

* test(search): cover FT.HYBRID RERANK placement

The test image has no disk-based indexes, so the spec checks the server
parses RERANK in the method params and only refuses it for the index.

* test(csc): pin eligible commands for 8.12 snapshot

The regenerated command snapshot makes BITRANGEINTARRAY, SEXPIRETIME
and SPEXPIRETIME cacheable. Update the pinned list and fingerprint.

* test(search): gate RERANK spec to Redis 8.12

Redis 8.10.3 rejects RERANK in KNN as an unknown argument. Note the
version range in the field docs.

* refactor(bitmap): align BitRangeIntArray with HLD

Follow the 8.12 BITRANGEINTARRAY client HLD: one method taking
*BitRangeIntArrayOptions, an optional Limit so LIMIT 0 can be sent,
and no client-side unit check (the server rejects bad units). Cover
the HLD tables, paging in BIT and BYTE units, and native bitmaps.

* feat(csc): cache BITRANGEINTARRAY in cluster mode

An array reply proves the key exists only when it is non-empty, so use
the same reply class as LRANGE.

* refactor(bitmap): split BitRangeIntArray options form

Keep BitRangeIntArray(ctx, key, start, end) for the plain call and
add BitRangeIntArrayWithOptions for the unit, limit and ZEROS, as in
the other WithX command pairs.
mgravell added a commit to seredis/StackExchange.Redis that referenced this pull request Nov 13, 2026
…eyed routing tests, SER1xx experiment block (#3355)

* Make the dedicated subscription connection lazy (#3310)

The dedicated pub/sub connection (always under RESP2; under RESP3 unless
SharedSubscriptionConnection is enabled) was opened eagerly as part of
activating every endpoint, so a client that never subscribes to anything
still paid for a second socket per node, and a second reconnect per node
on every failover. Seen on AMR, whose options provider disables the
configuration channel: 3.4.x showed two RESP3 connections where 3.0.x
showed one.

ActivateServer now opens that connection only when there is something to
subscribe to straight away: a configuration channel, or subscriptions that
already exist (and may need to move to this endpoint). Otherwise it is
created on first use, via the existing GetBridge paths; with the default
backlog policy the first SUBSCRIBE queues until it is up.

The connect wait ("second leg") and the maintenance handoff deadline
treated an absent subscription bridge as a disconnected one, which would
have left connects hanging; they now consider a connection that has not
been asked for as not required. The endpoint-state connect log says
"NotRequired" for that case instead of "Disconnected".

Tests: AzureManagedRedisConnectionCount now asserts one connection before
pub/sub and the expected count after; a new test covers the eager case
when a configuration channel is set; two tests that relied on the eager
subscription connection now subscribe first.

* Command coverage ledger and audit skill (#3311)

* Command coverage ledger and audit skill

Adds docs/CommandCoverage.md: every upstream Redis command the library does
not expose as a typed API, grouped and with a recorded decision (policy,
not applicable, deferred, todo, deprecated, enum-only, implemented). Seeded
from a diff of redis/redis src/commands (477 specs, @6a5fee67) against the
RedisCommand enum and source; it carries forward the dispositions from
#2055 (blocking is policy, Functions and ACL are deferred) and replaces
that hand-maintained list.

Adds the audit-command-coverage skill and its script, which re-runs the
diff on request and reports only what changed: new upstream commands with
no decision, ledger rows the code now covers, rows the spec no longer has,
and enum members the spec does not know. The script exits non-zero when
the ledger needs editing.

Closes #2055.

* Ledger: CLIENT TRACKING is v4, cluster/COMMAND introspection is administration, CLUSTER SHARDS note corrected

* Ledger: CLIENT PAUSE/UNPAUSE are operator tooling, not applicable

* Update RedisType.Bitmap and the keyspace-event tokens to reflect current server values (#3312)

* Object-type metadata for native bitmaps (redis/redis#15331), plus the tokens that diff turned up

RedisType gains Bitmap ("bitmap", the TYPE reply for a native roaring
bitmap, 8.12) and Gcra ("gcra", already in the server's type-name table).
KeyNotificationType gains SetBit (emitted under class $ for string bitmaps
and class b for native ones) and the other event names the server emits
that the enum lacked: arinsert, armset, arring, arseek, gcra, pfadd,
sexpire, spersist, sinterstore, zrangestore, and keymiss (class m, side
effect group).

Fixes the Copy token: the server has only ever emitted "copy_to", so the
"copy" token never parsed and a RedisChannel.KeyEvent(KeyNotificationType.Copy)
subscription never fired. The member and its value are unchanged.

Both parsers are generated from [AsciiHash]; tests extend the token tables.

* Drop Gcra: the type was withdrawn before release, so there is nothing for the token to match

* Add sunionstore/sdiffstore events (from the keyspace-notifications docs); drop the history comment on Copy

* Fix ConnectionGroupMember string-ctor dropping the name (#3314)

* Forward the name argument in ConnectionGroupMember's string constructor

The string overload chained to the ConfigurationOptions constructor without passing `name`, so every
member built from a configuration string (the idiom used throughout docs/Failover.md) was named after
its first endpoint by Init() instead of the name the caller gave it. Regression test added.

* Parenthesize a Debug-only assertion that trips SA1407

`endSegmentLength == 0 & endSegmentCommitted == 0` in DebugAssertValid sits inside `#if DEBUG`, so CI
(Release) never compiles it and never saw the StyleCop error; any local Debug build of RESPite failed
with TreatWarningsAsErrors. Introduced by ee47e53 after the StyleCop bump in aa1e5c0.

* Parse unbracketed IPv6 ip:port from the server (#1422) (#3307)

* Parse unbracketed IPv6 ip:port from the server (#1422)

CLUSTER NODES, MOVED/ASK and CLIENT LIST write IPv6 addresses without
brackets, i.e. "2a06:4e80:3:a::a:7000". The RFC 3986 parser used for
configuration strings reads that as a (valid!) eight-group address with
no port, so cluster nodes ended up as "[2a06:4e80:3:a::a:7000]:0" and
redirections and client addresses were equally wrong.

Add Format.TryParseServerEndPoint for the server's "ip:port" form: when
there is no ']' and more than one ':', split at the last colon and
require both halves to parse; anything else goes to the existing parser
unchanged, so configuration parsing is not touched. Use it at the three
sites that receive server-emitted text.

The sentinel +switch-master handler gets host and port as separate
tokens and was gluing them back together with ':' before parsing; pass
them to the (host, port) overload instead.

* Parse unbracketed IPv6 in MOVING/SMIGRATED endpoints

ParseMigrationEndPoint used the configuration-string parser, so an unbracketed IPv6 address parsed as an address with port 0, which the placeholder check then discarded. Use TryParseServerEndPoint, as the other server-emitted sites do; tests for the MOVING successor and SMIGRATED source/target.

* Profile commands sent in a batch (#2175) (#3313)

Co-authored-by: mohsinaliwork686-rgb <229973101+mohsinaliwork686-rgb@users.noreply.github.com>

* public-api.py: apply *REMOVED* at promotion; promote the API surface since 3.4.0 (#3316)

* public-api.py: apply *REMOVED* entries at promotion instead of copying them

The analyzer records a vanished shipped signature as a *REMOVED* line in
the unshipped file. Promotion moved every unshipped line across verbatim,
so a promotion after any such change would have left the shipped file
with both the dead signature and a *REMOVED* copy of it, and the analyzer
would then fail the build on the dead one (RS0017).

Promotion now consumes the marker: the named signature is dropped from the
shipped file, the marker is not carried over, and a marker that names
nothing shipped is an error. A *REMOVED* line already sitting in a shipped
file is also reported. The "+N promoted" note gains "-N removed", and an
unshipped file holding only removals is still emptied.

* Promote the shipped API surface: everything since 3.4.0

eng/public-api.py --promote. Fifty-two StackExchange.Redis entries (the
lag-aware availability surface, FailbackHealthCheck, HealthCheckProbeScope,
the 8.12-independent RedisType and KeyNotificationType tokens) and the
net6.0 SetUserPemCertificate overload move to shipped; the three parameter
renames from #3296 (muxer -> multiplexer on the Azure providers'
AfterConnectAsync, other -> obj on RESPite AsciiHash.Equals) are applied
rather than copied. Every unshipped file is now empty.

* Split third-party notices out of LICENSE so GitHub detects MIT (#3317)

* Split third-party notices out of LICENSE so GitHub detects MIT

The LICENSE file carried ~1.1KB of third-party notes after the MIT text,
which drops it below licensee's similarity threshold; GitHub reports the
repo license as "Other" (SPDX NOASSERTION). Keep LICENSE as the bare MIT
text and move the notes, verbatim, to THIRD-PARTY-NOTICES.md.

* Correct the third-party notices

- Redis: BSD-3 only through 7.2; RSALv2/SSPLv1 from 7.4, AGPLv3 added in
  8.0; point at redis.io/legal/licenses (the old /topics/license URL now
  redirects to the docs landing page).
- redis-doc: repo is archived under redis/redis-doc; note that redis/docs,
  its successor, is CC BY-NC-SA for newer content.
- Drop the Redis-64 and BookSleeve notes: neither is part of the
  development solution any more.
- Add the code actually incorporated from other projects (dotnet/runtime,
  dotnet/corefx, aspnet/BasicMiddleware) and the test-only redis-sharp copy,
  none of which the old notes mentioned.

* The read loop stops matching replies once teardown owns the response queue; a close with commands in flight no longer completes a call with the previous call's reply (#3330)

RecordConnectionFailed drains _writtenAwaitingResponse one message at a
time, completing each outside the lock, while the read loop can still be
matching replies that were already in the receive buffer. Each reply the
loop matched after the drain had taken a message went to the message
*after* the one it answered: GET key N completed, successfully, with key
N-1's value, and every reply behind it was off by one. Reproduced on
main with the reporter's loop: 616 wrong replies in 20,000 calls.

_isShutdown was already set under the queue's lock before the drain, and
the loop already consulted it - but only once the queue was empty.
Checking it before the dequeue closes the race: a reply arriving after
teardown has begun is dropped with the connection, and the message it
answered is failed by the drain as before.

The test re-creates the scenario against the real server (20 rounds of
500 in-flight GETs closed with allowCommandsToComplete: false) and
asserts that no call completes with another call's value; without the
fix it sees ~200 in a run.

Fixes #3329

* CLUSTER SLOT-STATS (8.2): per-slot usage statistics on IServer, experimental SER012 (#3341)

* CLUSTER SLOT-STATS (8.2): per-slot usage statistics on IServer, experimental SER012

IServer.ClusterSlotStatsBySlotRange(Async) and ClusterSlotStatsOrderBy(Async), modelled
on the CLUSTER SLOTS file layout and the HOTKEYS field handling: node-local read, lenient
nominal parsing, config-gated metrics as nullable properties (absent != zero), unknown
metrics preserved as Metadata, cpu-usec surfaced as TimeSpan. RedisFeatures.ClusterSlotStats
gates on 8.2 RC1. Promotes the CommandCoverage 'deferred' row.

* The storage-kind tripwire acknowledges SLOT-STATS and SLOTSRANGE: two new >8-byte literals, 28 -> 30

* Check the maximum operation count in HeartbeatConsistencyCheckPingsAsync (#3342)

* In-repo links follow the org move to seredis (#3326): github.com/StackExchange/StackExchange.Redis -> github.com/seredis/StackExchange.Redis everywhere, including the nupkg RepositoryUrl/PackageReleaseNotes metadata (#3344)

* MOVING handoff: hold new work in the backlog while the in-flight commands drain, so the recycle happens within a round trip instead of at the window end (#3347)

* MOVING handoff: hold new work in the backlog while the in-flight commands drain, so the recycle happens within a round trip instead of at the window end

The drain waited for "no caller work", which under steady traffic never comes: there is always a command in flight, so the recycle waited out the whole announced window and failed whatever was in flight at that instant. Measured live (endpoint rebind, single proxy, ~900 ops/s): recycle at +20.0s against a MOVING at +6.0s, 16 failed commands, missed-deadline warning.

While a handoff drains, the bridge holds a caller's work in the backlog instead of writing it to the connection about to be replaced; the backlog processor does not flush while the hold stands; the drain waits only for what is already written, capped at half the relaxed timeout (bounded by half the remaining window, floor one second); the connection is then recycled as before, the hold cleared, and the held work flushed to the replacement by the ordinary reconnect path. Timeouts are relaxed for the hold and the reconnect. A FailFast policy does not abandon held work for the handoff's own disconnect. Same scenario on this branch: recycle at +6.2s against a MOVING at +5.9s, 0 failed of 97k.

Deliberately a flag and nothing more: no per-message timeout cover (with relaxation off and a timeout shorter than the hold, held commands can time out in the backlog, as on any reconnect), and no retire under the write lock (a writer that reaches the lock after the recycle gets a write fault, as before). Also: the drain budget is recomputed after the DNS poll, a second MOVING during a handoff updates the successor pin rather than being dropped, and the pin lasts the window so the second bridge dials the same successor.

Tests: steady load (with and without relaxation), slow command inside the cap, blocking command past the cap; plus a fault-injector tier measurement that records the live timeline.

* OneLogicalEventFromEveryNodeRaisesOneEvent: wait for the one event rather than counting at the instant the second node relaxed; the callback is dispatched after the relaxation, so the count could be read as zero (flaked on main at ~1 in 6)

* INCREX and ARGREP are routed by key on a cluster (#3352)

IncrexMessageBase (#3065) and ArrayGrepMessage (#3076) derived from Message
directly, held a key, and never overrode GetHashSlot, so they reported
NoSlot. On a cluster that sends the command to an arbitrary node and leans
on -MOVED: an extra round trip for most calls, and a RedisServerException
under CommandFlags.NoRedirect.

Both now derive from Message.CommandKeyBase, as their single-key siblings
do, which supplies the slot (and the key in CommandAndKey).

Affects the StringIncrement overloads that take an Expiration (INCREX) and
ArrayGrep. No public API change.

---------

Co-authored-by: Mohsin Ali <mohsin.ali.work686@gmail.com>
Co-authored-by: mohsinaliwork686-rgb <229973101+mohsinaliwork686-rgb@users.noreply.github.com>
Co-authored-by: Nikolay Zdravkov <unsafeptr@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-notes indication that this issue needs to be mentioned in the release notes state:needs-doc-pr requires a PR to redis-doc repository

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

[NEW] Compressed bitmaps on Redis

8 participants