Visitar URL original
[BREAKING] Fix: wrong range query results for string bounds with astral characters or more code units than maxLength by pubkey · Pull Request #9554 · pubkey/rxdb · GitHub
Skip to content

[BREAKING] Fix: wrong range query results for string bounds with astral characters or more code units than maxLength - #9554

Open
pubkey wants to merge 1 commit into
masterfrom
claude/vigilant-newton-8rqsab-astral-characters-break-index-string-alignment
Open

pubkey wants to merge 1 commit into
masterfrom
claude/vigilant-newton-8rqsab-astral-characters-break-index-string-alignment

Conversation

@pubkey

@pubkey pubkey commented Nov 11, 2026

Copy link
Copy Markdown
Owner

This PR contains:

  • A BUGFIX
  • A BREAKING CHANGE

Describe the problem you have without this PR

range queries on indexed string fields returned wrong documents on the storages with custom index strings (memory, localstorage, and others) when the queried value contains astral characters like emojis or is longer than the maxLength of the field. The stored index strings are not changed.

Stored index strings are unchanged.

Breaking: count() with a string bound that contains a surrogate pair (emoji) or is longer than maxLength is now a slow count and throws QU14 without allowSlowCount: true.

Not part of this PR: the sort order when one over-long value is a prefix of another, and compound indexes where a later field has its own bound.

Changed files:

  • orga/changelog/fix-astral-characters-index-string-range-queries.json
  • src/custom-index.ts
  • src/query-planner.ts
  • test/unit/rx-storage-query-correctness.test.ts

Test output without the fix

The changed test file run on the memory storage with src/ from master:

  11 passing (461ms)
  1 failing

  1) rx-storage-query-correctness.test.ts
       range queries on an indexed string field with values and bounds that have more UTF-16 code units than maxLength:

      AssertionError [ERR_ASSERTION]: Expected values to be strictly deep-equal:
+ actual - expected

  [
+   'aa',
    'mm',
    'yy',
    'zz'
  ]

      + expected - actual

       [
      -  "aa"
         "mm"
         "yy"
         "zz"
       ]
      
      at _loop (./test_tmp/unit/rx-storage-query-correctness.test.js:184:18)
      at async Context.<anonymous> (./test_tmp/unit/rx-storage-query-correctness.test.js:233:13)

Test output with the fix

29 passing (1s)

Todos

  • Tests
  • Changelog

Assisted by

YES, CLAUDE CODE


Generated by Claude Code

@pubkey
pubkey force-pushed the claude/vigilant-newton-8rqsab-astral-characters-break-index-string-alignment branch from 3670d3a to af84d83 Compare November 12, 2026 12:10
@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Verify Test Reproduction: Tests PASSED without the fix (unexpected)

The changed tests do not fail without the source changes from this PR. Please inspect whether the test changes actually test the bug that the source changes fix.

This workflow runs the changed tests without the source fix to verify they reproduce the bug.

Show output
...(truncated, showing last 200 of 4593 lines)
      �[32m✓ �[39mshould NOT match non-compressible types
      �[32m✓ �[39mshould match types that include RFC 2045 parameters like charset
    MIME type preservation
      �[32m✓ �[39mfull roundtrip through storage should preserve MIME type on getData()
      �[32m✓ �[39mfull roundtrip should preserve MIME type for non-compressible type

  attachments-compression.test.ts (mode: gzip)
    basics
      �[32m✓ �[39mcompress->decompress
    CRUD
      �[32m✓ �[39mshould insert one attachment
      �[32m✓ �[39mshould get the attachment
      �[32m✓ �[39mshould get the data
    compare size
      �[32m✓ �[39mshould have a smaller size when compression is used
    selective compression
      �[32m✓ �[39mshould compress a compressible type (text/plain) and roundtrip correctly
      �[32m✓ �[39mshould NOT compress a non-compressible type (image/jpeg) but still roundtrip correctly
      �[32m✓ �[39mshould compress text but not jpeg on the same document
    isCompressibleType()
      �[32m✓ �[39mshould match wildcard patterns
      �[32m✓ �[39mshould match exact patterns
      �[32m✓ �[39mshould be case-insensitive
      �[32m✓ �[39mshould NOT match non-compressible types
      �[32m✓ �[39mshould match types that include RFC 2045 parameters like charset
    MIME type preservation
      �[32m✓ �[39mfull roundtrip through storage should preserve MIME type on getData()
      �[32m✓ �[39mfull roundtrip should preserve MIME type for non-compressible type
WARN LOG: �[36m'-------------- RxDB Open Core RxStorage -------------------------------
You are using the free Dexie.js based RxStorage implementation from RxDB https://rxdb.info/rx-storage-dexie.html?console=dexie 
While this is a great option, we want to let you know that there are faster storage solutions available in our premium plugins.
For professional users and production environments, we highly recommend considering these premium options to enhance performance and reliability.
 https://rxdb.info/premium/?console=dexie 
If you already purchased premium access you can disable this log by calling the setPremiumFlag() function from rxdb-premium/plugins/shared.
---------------------------------------------------------------------'�[39m

  migration-storage.test.ts (prev-major to newest (dexie))
    basic migrations
      �[32m✓ �[39mcreate both databases
      �[32m✓ �[39mshould migrate all documents
      �[32m✓ �[39mshould migrate in parallel
      �[32m✓ �[39mmigrate new->new should also work
    issues
      �[32m✓ �[39mmigration with multiple collections

  migration-storage.test.ts (newest to newest)
    basic migrations
      �[32m✓ �[39mcreate both databases
      �[32m✓ �[39mshould migrate all documents
      �[32m✓ �[39mshould migrate in parallel
      �[32m✓ �[39mmigrate new->new should also work
    issues
      �[32m✓ �[39mmigration with multiple collections

  webmcp.test.ts
    �[32m✓ �[39mshould register query tool when registerWebMCP is called
    �[32m✓ �[39mshould wait for changes using wait_changes tool
    �[32m✓ �[39mchanges tool should return documents without internal meta fields
    �[32m✓ �[39mshould iterate over changes using checkpoint
    �[32m✓ �[39mshould execute modifier tools successfully (insert/upsert/delete)
    �[32m✓ �[39mshould unregister tools when collection is closed
    �[32m✓ �[39mshould not register modifier tools when readOnly is true
    �[32m✓ �[39mshould emit log$ and error$ events for executed tools
    �[32m✓ �[39mshould register tools for newly added collections dynamically
    custom targets
      �[32m✓ �[39mshould build the same tools for a target that is not an RxCollection
      �[32m✓ �[39mshould throw WMCP1 from a target that has no such document
      �[32m✓ �[39mshould not build modifier tools for a readOnly target
      �[32m✓ �[39mshould respect awaitReplicationsInSync on a target
      �[32m✓ �[39mshould register at a given modelContext and unregister on close
      �[32m✓ �[39mshould emit log$ and error$ for a target
      �[32m✓ �[39mshould build a working target from an RxCollection

  crdt.test.ts
    collection creation
      �[32m✓ �[39mshould throw if the wrong conflict handler is set
      �[32m✓ �[39mshould automatically set the CRDT conflict handler
    .insert()
      �[32m✓ �[39mshould insert a document and initialize the crdt state
      �[32m✓ �[39mshould insert document via bulkInsert
    .insertCRDT()
      �[32m✓ �[39mshould insert the document
      �[32m✓ �[39mshould insert the document with undefined argument
      �[32m✓ �[39mshould respect the if-else logic
    .remove()
      �[32m✓ �[39mshould delete the document via .remove
    .incrementalPatch()
      �[32m✓ �[39mshould update the document
    disallowed methods
      �[32m✓ �[39mshould throw on incrementalModify
      �[32m✓ �[39mshould throw on modify
    redirected methods
      �[32m✓ �[39mshould redirect patch through updateCRDT
      �[32m✓ �[39mshould redirect incrementalRemove through updateCRDT
      �[32m✓ �[39mshould redirect update through updateCRDT
    conflict handling
      init
        �[32m✓ �[39minit
      .getCRDTConflictHandler()
        �[32m✓ �[39mshould merge 2 inserts correctly
        �[32m✓ �[39mshould preserve schema default values during conflict resolution
        �[32m✓ �[39mshould preserve the composite primary key during conflict resolution
      conflicts during replication
        �[32m✓ �[39mshould merge the +1 increments

  population.test.js
    createRxSchema
      positive
        �[32m✓ �[39mshould allow to create a schema with a relation
        �[32m✓ �[39mshould allow primary as relation key
        �[32m✓ �[39mshould allow to create a schema with a relation in nested
        �[32m✓ �[39mshould allow to create relation of array
        �[32m✓ �[39mshould allow to create relation with nullable string
      negative
        �[32m✓ �[39mthrow if ref-type is no string
        �[32m✓ �[39mthrow if ref-type is no string (array)
    RxDocument().populate()
      positive
        �[32m✓ �[39mpopulate top-level-field
        �[32m✓ �[39mpopulate nested field
        �[32m✓ �[39mpopulate string-array
        �[32m✓ �[39mpopulate with primary as ref
      negative
        �[32m✓ �[39mthrow DOC5 for a path that does not exist in the schema, even when the value is falsy
        �[32m✓ �[39mthrow DOC6 when populating a non-ref schema field, even when the value is falsy
    RxDocument populate via pseudo-proxy
      positive
        �[32m✓ �[39mpopulate top-level-field
        �[32m✓ �[39mpopulate nested field
    issues
      �[32m✓ �[39m#222 population not working when multiInstance: false
      �[32m✓ �[39mpopulate array should preserve the order of ref ids when two documents reference the same set in different order
      �[32m✓ �[39mpopulate array when ref is defined on items instead of on the array field

  leader-election.test.js
    .die()
      �[32m✓ �[39mother instance applies on death of leader
    election
      �[32m✓ �[39ma single instance should always elect itself as leader
      �[32m✓ �[39mshould not elect as leader if other instance is leader
      �[32m✓ �[39mwhen 2 instances apply at the same time, one should win
      �[32m✓ �[39mwhen many instances apply, one should win
      �[32m✓ �[39mwhen the leader dies, a new one should be elected
    cleanup
      �[32m✓ �[39mshould properly call die() on the elector when the database is closed
      �[32m✓ �[39m#8893 close() must not resolve before the broadcast channel is closed
    integration
      �[32m✓ �[39mnon-multiInstance should always be leader
      �[32m✓ �[39mnon-multiInstance: waitForLeadership should instant
      �[32m✓ �[39mwaitForLeadership: run once when instance becomes leader

  import-export.test.js
    Collection
      .exportJSON()
        �[32m✓ �[39mexport the collection
        �[32m✓ �[39mexport encrypted as decrypted
      .importJSON()
        positive
          �[32m✓ �[39mimport json
        negative
          �[32m✓ �[39mshould not import if schema is different
    Database
      .exportJSON()
        �[32m✓ �[39mshould export a valid dump
        �[32m✓ �[39mexport encrypted as decrypted
        �[32m✓ �[39mexport with multiple collections
        �[32m✓ �[39mexport 1 of 2 collections
      .importJSON()
        positive
          �[32m✓ �[39mimport dump
        negative
          �[32m✓ �[39mshould not import if schema is different
    issues
      �[32m✓ �[39m#319 collections must be created before importDump
      �[32m✓ �[39m#1396 import/export should work with attachments
    attachments
      �[32m✓ �[39mshould not contain the attachments by default
      �[32m✓ �[39mshould contain the attachments data as base64
      �[32m✓ �[39mshould restore the attachments on import
      �[32m✓ �[39mshould work on the database level

  database-lifecycle.ts
    �[32m✓ �[39mdo some writes updates and deletes and cleanups and reopens

  last.test.ts (dexie)
    �[32m✓ �[39mrun a minimal performance test to ensure the performance function works
    �[32m✓ �[39mensure all Memory RxStorage instances are closed
    �[32m✓ �[39mensure every db is cleaned up
    �[32m✓ �[39mensure all collections are closed
    �[32m✓ �[39mensure all BroadcastChannels are closed
    �[32m✓ �[39mensure all replication states are closed
    �[32m✓ �[39mensure all RemoteMessageChannels have been closed
    �[32m✓ �[39mensure all websockets have been closed
    �[32m✓ �[39mensure all leader electors are dead
    �[32m✓ �[39mexit the process

Chrome Headless 156.0.0.0 (Linux 0.0.0): Executed 1448 of 1448�[32m SUCCESS�[39m (1 min 0.892 secs / 59.984 secs)
�[32mTOTAL: 1448 SUCCESS�[39m



View full workflow run

…rs or more code units than maxLength

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015P6w6hnnyVPXwotts6YQ2F
@pubkey
pubkey force-pushed the claude/vigilant-newton-8rqsab-astral-characters-break-index-string-alignment branch from af84d83 to bf1d233 Compare November 13, 2026 13:16

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants