Going beyond 2 billion vectors #594

Activity

1. jshook commented on Jan 7, 2026

   

   jshook

   on Jan 7, 2026

   Contributor

   More actions

   I've hit this barrier before in several places and have a strong perspective to share.
   I'll cut to the chase:
   64-bit twos-complement (i.e. Java long) is the most optimal form for us to use here, and any concerns about saving space should be deferred into serdes or other non-runtime areas.

   Why?

   1. It is the natural unit of native atomic memory access on all modern x86 variants.
   2. When dealing with dense in-memory structures, Boxing is evil. Boxing and hashmaps are two of the most under-recognized heap padding conveniences.
   3. The runtime representation and the externalized representation of indices, etc can be different. Most of the time complexity in managing the mapping between these layers is tied up in IO for most cases, notwithstanding other unavoidable requirements.
   4. Maintaining dual-mode logic is very very risky, particularly when it comes to structure and traversal logic. We should pick what will work and invest in it to make it robust, and correct, and then optimize where needed for efficiency. We can't easily do that if we have two different mental models of indexing.

   I'm not entirely convinced that we have to scale test up to the magnitude of the maximum representable value. Good boundary testing will cover 90% of the worry for what "could" be represented.

   There are other reasons to add to this, but this should be enough to discuss.
2. MarkWolters commented on Oct 21, 2026

   

   MarkWolters

   on Oct 21, 2026

   Contributor

   More actions

   Why the 2B limit doesn't bite in practice:

   1. Integrators already partition the data, and their own IDs are 32-bit. Your point about Cassandra holds, and it's stronger than "nobody builds indexes that big." Cassandra SAI builds one jvector index per SSTable segment and addresses rows in a segment with int segment row IDs. So Cassandra couldn't hand a single jvector index more than 2^31 ordinals even if it wanted to. Lucene-based integrations, such as the OpenSearch jvector plugin, have the same limit, because Lucene doc IDs are ints and a segment is capped at about 2^31 docs. jvector being int-based matches the systems it's built for.

   2. Other limits hit well before 2B. At 1B vectors × 768 dims you have about 3 TB of raw floats. The level-0 adjacency at M=32 is about 128 GB of int arrays on the heap. That graph isn't buildable on-heap with the current build pipeline. Building, compacting and merging at that size are infeasible long before ordinals run out.

   3. Switching to long is expensive, and the cost lands on the hot path.

   • NodeQueue packs the score and node ID into one long: sortable-float score in the high 32 bits, node in the low 32 (NodeQueue.java:125-128). That packing is what makes the candidate and result heaps cheap.
     With long ordinals it no longer fits. You'd need parallel arrays or a slower heap, right in the search inner loop.
   • Adjacency lists are the biggest memory cost, both on-heap and on disk. Long ordinals double them and halve how many neighbors fit in cache. Every graph user would pay this to serve a hypothetical one.
   • It needs a new on-disk format, and the fused PQ/ASH layouts, compactor, ordinal mappers and native kernels would all change.