Visitar URL original
Peel xor filters through one queue per segment by tisonkun · Pull Request #786 · apache/datasketches-java · GitHub
Skip to content

Peel xor filters through one queue per segment - #786

Open
tisonkun wants to merge 1 commit into
apache:mainfrom
tisonkun:xorfilter-segment-queues
Open

tisonkun wants to merge 1 commit into
apache:mainfrom
tisonkun:xorfilter-segment-queues

Conversation

@tisonkun

Copy link
Copy Markdown
Member

Problem

XorFilter construction peels singleton slots through a single LIFO queue that spans all three segments. The reference C implementation (FastFilter xor_singleheader xor8_populate), the Rust xorf crate, and the datasketches-rust implementation in apache/datasketches-rust#309 instead keep one queue per segment and drain segment 0, then 1, then 2, repeating until all queues are empty.

Whether peeling succeeds does not depend on the order, because the 2-core of the hypergraph is unique. Both orders therefore select the same construction seed and write the same header. The order does decide which slot each key owns, though, so the fingerprint payloads differ. For the same items and base seed, Java and Rust produced different images, and a different set of non-member keys tested as false positives.

Change

  • map() splits the existing queue array into one region per segment and peels in the reference order. A slot is queued at most once, so each region of segmentLength entries cannot overflow, and no memory is added.
  • A peeled key is removed only from its slots in the other two segments. Its own slot holds no other key, so the slot is never queued again.
  • Queue lengths stay in local variables so the JIT can keep them in registers instead of reloading them from an array that may alias the other int[] buffers.
  • referenceImplementationImageTest checks 8- and 16-bit images against bytes produced by datasketches-rust for the same items and base seed. The test fails with the previous peeling order.

Compatibility

The serialized format does not change. Existing images remain readable and answer every query exactly as before, because queries depend only on the stored seed, segment length, and fingerprints. Only newly built images differ. XorFilter has not been released yet.

Verification

  • XorFilterTest: 25 tests pass.
  • Cross-implementation check against datasketches-rust (feat(xor): add immutable filters with portable serialization datasketches-rust#309): 8- and 16-bit filters, base seeds 42 and 7, and 4, 100, 1,000, 16,384, 100,000, and 1,000,000 items (24 configurations). All images are byte-identical, and each library returns identical answers on the other's images for every member plus 200,000 non-members.
  • Construction time, measured through the package-private constructor on pre-generated distinct keys, so item hashing and deduplication are excluded. Each figure is the median of six JVM runs, alternating old and new builds; each run reports the median of seven timed rounds after warm-up. JDK 25.0.2, Apple M4 Max.
Workload Single queue Segment queues Change
1,000 filters of 16,384 keys 343.6 ms 346.1 ms +0.7%
5 filters of 1,000,000 keys 158.3 ms 129.1 ms -18.4%
1 filter of 8,000,000 keys 746.9 ms 476.1 ms -36.3%

The gain grows once the working set exceeds the CPU caches. Draining one segment at a time keeps its reads close to sequential order, because peeling a key only creates new singletons in the other two segments. With a single LIFO queue, each new singleton is popped immediately, which follows chains of slots across the whole table.

XorFilter construction peeled singleton slots through one LIFO queue over
all slots. The reference C implementation (FastFilter xor_singleheader),
the Rust xorf crate, and datasketches-rust keep one queue per segment and
drain segment 0, then 1, then 2. Whether peeling succeeds does not depend
on the order, so every implementation selects the same construction seed,
but the order decides which slot each key owns. The same items and base
seed therefore produced different fingerprint payloads in Java and Rust,
and different non-member keys were false positives.

Peel through one queue region per segment within the existing queue
array, in the reference order. A peeled key is removed only from its slots
in the other two segments, because its own slot holds no other key. Images
now match datasketches-rust byte for byte, and draining one segment at a
time keeps its reads close to sequential order, which speeds up
construction of large filters. The serialized format is unchanged, so
existing images remain readable and answer queries as before.
@tisonkun
tisonkun requested review from freakyzoidberg and leerho and a balanced review from Copilot October 30, 2026 03:21

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

private static byte[] fromHex(final String hex) {
final byte[] bytes = new byte[hex.length() / 2];
for (int i = 0; i < bytes.length; ++i) {
bytes[i] = (byte) Integer.parseInt(hex.substring(2 * i, (2 * i) + 2), 16);

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants