Repository navigation
Conversation
XorFilter construction peeled singleton slots through one LIFO queue over all slots. The reference C implementation (FastFilter xor_singleheader), the Rust xorf crate, and datasketches-rust keep one queue per segment and drain segment 0, then 1, then 2. Whether peeling succeeds does not depend on the order, so every implementation selects the same construction seed, but the order decides which slot each key owns. The same items and base seed therefore produced different fingerprint payloads in Java and Rust, and different non-member keys were false positives. Peel through one queue region per segment within the existing queue array, in the reference order. A peeled key is removed only from its slots in the other two segments, because its own slot holds no other key. Images now match datasketches-rust byte for byte, and draining one segment at a time keeps its reads close to sequential order, which speeds up construction of large filters. The serialized format is unchanged, so existing images remain readable and answer queries as before.
tisonkun
requested review from
freakyzoidberg and
leerho
and
a balanced review from Copilot
October 30, 2026 03:21
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
XorFilterconstruction peels singleton slots through a single LIFO queue that spans all three segments. The reference C implementation (FastFilter xor_singleheaderxor8_populate), the Rust xorf crate, and the datasketches-rust implementation in apache/datasketches-rust#309 instead keep one queue per segment and drain segment 0, then 1, then 2, repeating until all queues are empty.Whether peeling succeeds does not depend on the order, because the 2-core of the hypergraph is unique. Both orders therefore select the same construction seed and write the same header. The order does decide which slot each key owns, though, so the fingerprint payloads differ. For the same items and base seed, Java and Rust produced different images, and a different set of non-member keys tested as false positives.
Change
map()splits the existingqueuearray into one region per segment and peels in the reference order. A slot is queued at most once, so each region ofsegmentLengthentries cannot overflow, and no memory is added.int[]buffers.referenceImplementationImageTestchecks 8- and 16-bit images against bytes produced by datasketches-rust for the same items and base seed. The test fails with the previous peeling order.Compatibility
The serialized format does not change. Existing images remain readable and answer every query exactly as before, because queries depend only on the stored seed, segment length, and fingerprints. Only newly built images differ.
XorFilterhas not been released yet.Verification
XorFilterTest: 25 tests pass.The gain grows once the working set exceeds the CPU caches. Draining one segment at a time keeps its reads close to sequential order, because peeling a key only creates new singletons in the other two segments. With a single LIFO queue, each new singleton is popped immediately, which follows chains of slots across the whole table.